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Abstract 

The  goal  of  this  research  effort  was  to  determine  recom 
mended  topics  to  be  taught  in  the  Air  Force  Institute  of 
Technology  Professional  Continuing  Education  course  CCN  331 
Executive  Cost  and  Price  Analysis.  This  determination  of 
course  content  was  necessary  due  to  the  changes  required  by 
the  implementation  of  Department  of  Defense  Directive 
5000. 52-M,  "Career  Development  Program  for  Acquisition  Per¬ 
sonnel."  The  determination  was  accomplished  by  developing  a 
list  of  suggested  topics  and  surveying  a  panel  of  six  Govern 
ment  pricing  executives,  managers,  and  experts  via  the  Delph: 
method.  Kendall's  Coefficient  of  Concordance  test  was  uti¬ 
lized  to  determine  the  significance  of  agreement  between  the 
experts.  Using  this  methodology,  a  consensus  was  reached  on 
a  prioritized  list  of  25  topics.  In  addition,  recommenda¬ 
tions  for  specific  areas  of  study  relating  to  a  topic  wer: 
received  for  14  topics.  The  consensus  list  of  prioritized 
topics  and  the  recommended  subtopics  constitute  our  determi¬ 
nation  of  content  for  CON  331  -  Executive  Cost  and  Price 


Analysis . 


CONTENT  OF  AIR  FORCE  INSTITUTE  OF  TECHNOLOGY  COURSE  CON  331 


AS  DETERMINED  BY  EXPERTS  IN  THE  FIELD:  A  CONTRACTING  SURVEY 
ANALYSIS  USING  THE  DELPHI  TECHNIQUE 

Chapter  I 

Background  and  Statement  of  the  Problem 


General  Issue 

Since  World  War  II,  each  presidential  study  reviewing 
the  problems  within  the  field  of  acquisition  has  cited  the 
need  for  a  better  trained,  more  professional  workforce 
(Pursch  1992).  Specifically,  the  President's  Blue  Ribbon 
Commission  on  Defense  Management,  better  known  as  the  Packard 
Commission,  stated 

Federal  regulations  should  establish  business  related 
education  and  experience  criteria  for  civilian  contract¬ 
ing  personnel,  which  will  provide  a  basis  ror  the  pro¬ 
fessionalization  of  their  career  paths  i Presidents 
Blue  Ribbon  Commission  1986:27). 

To  address  these  concerns.  Congress  passed  Chapter  S7  of 
Title  10,  United  States  Code,  and  the  Department  of  Defense 
(DOD)  issued  DOD  Directive  5000.52,  "Defense  Acquisition 
Education,  Training,  and  Career  Development  Program",  and 
5000. 52-M,  "Career  Development  Program  for  Acquisition  Per¬ 
sonnel",  which  implemented  the  law  as  well  as  consolidated 
several  DOD  regulations  and  directives  relating  to  education 
for  acquisition  personnel  (DOD  1991:i).  Hence,  the  Undersec- 
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retary  of  Defense  for  Acquisition  has  directed  the  establish¬ 
ment  of  appropriate  education,  training,  and  experimental 
programs  and  prerequisites  that  will  enable  acquisition 
personnel  to  obtain  the  level  of  education  and  experience 
required  for  their  assignments  and  responsibilities. 

To  provide  appropriate  education,  training,  and  experi¬ 
ence,  each  specialty  within  the  acquisition  career  field 
(e.g.  program  management,  contracting,  etc.),  was  reviewed  by 
the  Department  of  Defense  and  specific  education  and  experi¬ 
ence  requirements  were  established  for  entry  (Level  I), 
journeyman  (Level  II),  and  managerial  and  executive  (Level 
III}  personnel  (DOD  1991:2-1).  Various  Air  Force  Institute 
of  Technology  (AFIT)  Professional  Continuing  Education  (FCZ,, 
departments  are  conducting  studies  to  determine  how  well  the 
PCE  courses  meet  the  needs  of  the  acquisition  professionals 
and  how  to  improve  the  various  courses. 

Specific  Problem 

In  order  for  an  acquisition  official  in  the  Pricing 
career  field  to  receive  Level  III  certification,  they  must 
successfully  complete  the  AFIT  PCE  course  CON  331  -  Executive 
Cost  and  Price  Analysis.  The  current  objective  of  the  course 
is  to  provide  the  journeyman  price  analyst  with  training  to 
improve  their  technical  skills  and  increase  their  technical 
level  of  competency.  Comments  from  pricing  managers  and 
executives  who  have  recently  taken  the  course  have  raised 
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concerns  regarding  the  usefulness  and  applicability  of  the 
course  as  currently  structured.  Therefore,  this  research 
addresses  the  following  problem:  Now  that  CON  331  is 
required  for  Level  III  certification  for  pricing  managers  and 
executives,  what  changes  in  the  course  curriculum  need  to  be 
made  to  meet  the  needs  of  current  and  future  contracting 
managers? 

Investigative  Questions 

In  order  to  resolve  this  specific  problem,  we  identified 
the  attributes  of  the  current  course  offering,  including 
personnel  who  are  required  to  take  the  course.  This  provided 
a  baseline  for  recommending  changes  to  the  course.  After 
this  information  was  identified  about  the  current  course, 
recommendations  as  to  what  topics  need  to  be  taught  in  future 
courses  were  provided.  This  thought  process  led  to  the 
following  specific  investigative  questions  that  were  an¬ 
swered  . 

1.  Who  is  presently  required  to  tai;e  the  course? 

2.  What  are  the  current  course  objectives  and 
content? 

3.  What  specific  topics  represent  the  needs  of  the 
people  who  are  required  to  take  the  course? 
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Scope  and  Limitations 


This  research  addressed  only  the  education  program  for 
pricing  executives  and  managers  in  the  contracting  career 
path  as  set  forth  in  DOD  5000. 52M  "Career  Development  Program 
for  Acquisition  Personnel". 

Definitions 

To  allow  for  a  common  basis  of  understanding,  defini¬ 
tions  of  the  following  terms  are  provided. 

Expert .  The  expert  is  one  who  has  a  knowledge  of  the 
problem  area,  has  a  good  performance  record  in  his  area  of 
expertise,  has  a  high  degree  of  objectivity  and  rationality 
(Tersine  and  Riggs,  1976:53-54),  is  broadly  representative  of 
the  population  that  is  being  assessed,  and,  using  Mary  Parker 
Follett's  (1924)  definition  of  expert:  "has  experience  in 
the  organization  and  who  therefore  knows  more  about  life 
inside  the  organization  than  anyone  on  the  outside"  (Cishfski 
and  Joseph,  1991:298). 

Pricing  Executive  or  Manager .  Any  individual  in  the 
contracting  career  field  (GS-1102  and  comparable  military) 
who  has  the  grade  of  GS/GM-13  or  above  or  the  rank  of  0-4  or 
above  and  hoids  a  job  that  is  "primarily  cost  and  price 
analysis  oriented"  (DoD  1991:  B.1-5). 

Course  Curricul urn.  Those  topics,  subtopics,  and  points 
of  emphasis  that  compromise  the  subject  matter  to  be  taught 
during  the  class  sessions. 
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Needed  Topics .  Areas  of  expertise  or  knowledge  required 
for  a  person  to  adequately  perform  the  job  of  a  pricing 
executive  or  manager. 

Pricer ,  Price  Anal yst ,  or  Cost  and  Price  Anal yst .  An 
individual  who  reviews  and  evaluates  "the  separate  cost 
elements  and  proposed  profit  of  (a)  an  offeror's  or  contrac¬ 
tor's  cost  or  pricing  data  and  (b)  the  judgmental  factors 
applied  in  projecting  from  the  data  to  the  estimated  costs  in 
order  to  form  an  opinion  on  the  degree  to  which  the  proposed 
costs  represent  what  the  cost  of  the  contract  should  be, 
assuming  reasonable  economy  and  'efficiency"  (43  CFR  15.301). 


Thesis  Overview 

This  thesis  is  composed  of  five  chapters.  This  chapter 
provided  a  detailed  statement  of  the  problem  of  revising  a 
training  course  for  pricing  executives  and  managers,  de¬ 
scribed  how  the  problem  arose,  and  provided  the  bac.iground 
information  associated  with  the  problem.  Chapter  II 
the  results  of  the  literature  review  conducted  in  order  to 
answer  investigative  questions  one  and  two  and  delineates  the 
thought  process  used  to  develop  the  method  for  selecting 
course  topics.  In  Chapter  III  we  explain  and  justify  the 
research  design  and  methodology  by  providing  in  detail  the 
rationale  for  using  the  selected  research  methodology  as  well 
as  describing  the  exact  procedures  used  throughout  the  re¬ 
search.  Chapter  IV  describes  the  results  obtained  by  follow- 
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ing  our  research  design  and  presents  an  analysis  of  the 
results.  Finally,  Chapter  V  summarizes  the  research  effor 
provides  conclusions  based  upon  the  research  results,  and 
describes  recommendations  for  future  studies  relating  to  t 
research. 


chapter  II 
Literature  Review 


Introduction 

In  this  chapter,  the  need  for  training  for  pricing 
executives  and  managers  is  explored.  The  population  current¬ 
ly  required  to  take  the  executive  pricing  course  is  described 
and  the  topics  covered  in  the  current  course  are  delineated. 
This  discussion  answers  the  first  two  investigative  questions 
(the  third  investigative  question  is  addressed  in  Chapter 
III).  Finally,  an  appropriate  method  for  selecting  topics 
for  an  academic  course  is  explained. 

Education  Requirement  Established 

Continuing  education  and  training  is  required  fcr  ail 
executive  and  management  personnel  if  they  are  to  remain 
effective  and  efficient  (Palmer  1972:  398;  Koval  1971:  51; 
Floyd  1990;  and  others).  The  President's  Blue  Ribbon  Commis - 
sion  on  Defense  Management  recommended  that  the  Department  cf 
Defense  provide  better  training  for  professionals  in  the 
acquisition  career  field  (President's  Blue  Ribbon  Commission, 
1986:27).  In  response,  the  Department  of  Defense  designated 
mandatory  Professional  Continuing  Education  (PCE)  courses  fcr 
each  of  three  levels  --  basic  level  (Level  I),  intermediate 
level  (Level  II),  and  senior  level  (Level  III)  personnel  -- 
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for  each  acquisition  career  field  (DCD  1991:2-1;  Carlton 
1989:  Chapter  I).  To  determine  how  well  the  AFIT  ?CE  course 
CON  331  meets  the  needs  of  the  personnel  required  to  take  the 
course,  investigative  questions  1  and  2  were  addressed  during 
the  literature  review. 

Investigative  Question  1.  :i::  Who  Is  Present  1  y  Required  to  Take 
the  Course? 

Level  III  certification  is  required  for  any  GS/GM  12  and 
above  or  04  and  above  in  the  acquisition  career  field  (DCD 
1991:2-1).  DOD  Directive  5000. 52M,  "Career  Development 
Program  for  Acquisition  Personnel,"  identified  the  acquisi¬ 
tion  career  field  to  include  the  following  specialties: 
program  management;  communications  computer  systems;  con¬ 
tracting  ;  purchasing;  industrial  property  management;  systems 
planning  research,  development,  and  engineering;  test  and 
evaluation;  manufacturing  and  production;  quality  asourunoe: 
acquisition  logistics;  business,  cost  estimating  and  finan¬ 
cial  management;  and  auditing  (USD(A)  1992:  ii). 

The  contracting  specialty  (GS-1102  and  comparable  raili- 
tary)  includes  the  title  designators  of  Contract  Negotiator, 
Contract  Specialist,  Contract  Termination  Specialist,  Con¬ 
tractor  Administrator,  Procurement  Analyst,  Administrative 
Contracting  Officer,  Contract  Price  and/or  Cost  Analyst . 
Contracting  Officer,  Procuring  Contracting  Officer,  and 
Termination  Contracting  Officer  (DOD  1991:  B.1-1). 
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WQMT  540 


--  "Advanced  Contract  Pricing"  has  been  desig¬ 
nated  as  a  training  roquirement  for  Level  III  certification 
for  Contract  Price  and/cr  Cost  Analysts  in  the  acquisition 
career  field  specialty  of  contracting.  The  Defense  Acquisi¬ 
tion  University  renamed  WQMT  540  to  CON  331  --  "Executive 
Cost  and  Price  Analysis"  (USD(A)  1992:  21). 

Accordingly,  to  obtain  certification,  any  individual  in 
the  acquisition  career  field  specialty  of  contracting  (C3- 
1102  and  comparable  military)  who  has  the  grade  of  GS/GM  13 
or  above  or  the  rank  of  0-4  or  above  and  holds  a  job  that  is 
"primarily  cost  and  price  analysis  oriented"  is  required  to 
have  completed  CON  331  --  "Executive  Cost  and  Price  Analysis 
(DOD  1991:  B.1-5)  . 

Investigative  Question  2  "  What  Are  the  Current  Course 
Objectives  and  Content? 

The  current  course  --  CON  331,  formerly  WQMT  540  --  was 
developed  in  the  early  1980 's  to  address  specific  problerr.s 
being  experienced  at  that  time  and  was  directed  toward 
"journeyman  working  analysts"  rather  tha.n  managers  and  execu 
tives  (Baneman  1993).  The  objectives  of  the  course,  as 
reflected  in  the  current  course  description  are  to  "Provide 
experienced  procurement  analysts  with  statistics  and  account 
ing  tools  necessary  to  analyze  costs,  establish  cost  estimat 
ing  relationships,  and  develop  reasonable  cost  estimates  in 
large  system  procurements"  (AFIT  1989:  201-202).  Topics  in 
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the  current  course  include  the  impact  of  unallowable  pod  and 
base  costs  on  overhead  allocation,  probability  and  decision 
analysis,  tracking  actual  costs  to  estimated  costs  to  ascer¬ 
tain  bias,  and  risk  analysis  and  production  quantity.  The 
course  was  not  originally  designed  to  meet  the  needs  of 
pricing  executives  and  managers. 

Need  to  Review  Curriculum. 

With  the  change  in  clientele  for  CON  331  due  to  the 
Level  III  certification  requirements  and  the  current  course 
objectives  and  content  structured  around  the  journeyman  price 
analyst,  comments  on  the  "End  of  Course  Critique"  reflected  a 
growing  dissatisfaction  in  the  course  content  which  was  .act 
evident  in  previous  years'  critiques  (Daneman,  1992).  Brown 
found  that  with  a  change  in  doctrine,  the  educational  prcgrom 
for  logistician  executives  needed  to  be  changed  to  meet  the 
new  requirements  (Brown,  1990).  Accordingly,  the  research  of 
this  thesis  is  focused  on  determining  the  topics  that  best 
fill  the  educational  needs  of  those  required  to  jjir.piete  :c:i 
331  for  Level  III  certification. 

An  Approach  for  Selecting  Topics  for  Course  Design 

A  thorough  review  of  available  literature  revealed 
nothing  that  set  forth  curriculum  to  train  pricing  executives 
and  managers.  Notwithstanding,  several  items  have  been 
written  for  entry  and  intermediate  level  pricers  (AFIT  1939; 
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Crumbie  and  Willis  1330),  as  well  as  for  executives  in  other 
closely  related  fields  such  as  accounting  and  business  aixin 
istration  (Palmer  1972:  393;  and  Reeve  and  Jensen  1972:  345) 
Some  sources  suggested  that  the  nature  of  such  a  course 
should  be  only  to  update  the  trainees  on  law  and  regulatory 
changes  (JFMIP  1990).  However,  that  purpose  is  fulfilled  by 
an  annual  seminar  required  of  all  pricing  chiefs  and  by 
another  mandatory  cour  e  contracting  curriculum,  CCN 

301  "E:<ecutive  Contracting"  (C£D(A)  1992:  21).  Accordingly, 
the  nature  of  a  pricing  course  for  managers  and  executives  i 
more  similar  to  a  collegiate  course  with  a  well  defined  and 
timely  curriculum  covering  specific  topics. 

Delphi  Discussion .  While  there  exists  an  abundance  or 
data  concerning  managerial  and  executive  training  in  general 
no  relevant  data  exists  concerning  specific,  topical  educa¬ 
tional  needs  for  pricing  executives  and  managers.  Consider¬ 
ing  the  extant  condition,  this  research  must  be  e;:pl  oratory 
in  nature.  Whenever  insufficient  or  no  applicable  data 
exists  the  Delphi  method  is  considered  an  appropriate  method 
ology  (Zaloom  1974:  32).  This  method  not  only  permits  the 
gathering  of  information,  but  also  provides  for  information 
to  be  gathered  from  the  people  affected  by  the  results,  the 
"customer".  This  customer  input  is,  in  our  opinion,  vital 
and  though  highly  touted  by  the  professional  educational 
establishment,  is  rarely  used  in  curriculum  development. 
Prior  research  on  the  use  of  the  Delphi  method  in  education 
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needs  assessment  and  curriculum  design  support  the  use  of  no 
more  than  three  iterations,  and  also  supporr  the  use  of  both 
a  Likert  scale  and  a  ranking  process  to  establish  the  rela¬ 
tive  importance  of  the  various  topics.  (Clshfski  and  Joseph 
1991:  297;  Weaver  1988:  18;  Martin  and  Chaney  1993:  267). 
Based  on  this  prior  research,  the  Delphi  technique  was  used 
to  answer  investigative  question  3  as  described  in  the  next 
chapter.  A  more  detailed  discussion  of  the  Delphi  method  is 
provided  in  Appendix  A. 

Summary 

This  chapter  reviewed  pertinent  literature  regarding  the 
requirement  for  executive  education  in  general  and  the  DCD's 
"Executive  Cost  and  Price  Analysis"  Course.  Two  of  the 
investigative  questions  were  answered:  Who  is  required  to 
take  the  course?  and  What  topics  are  covered  in  the  current 
course?  Finally,  based  on  the  literature  review,  the  Deliih:. 
method  was  selected  as  the  appropriate  methodology  to  det-..- 
mine  the  topics  needed  for  CON  331.  In  the  next  chapter,  t'.e 
methodology  used  in  selecting  the  required  topics  is  dis¬ 
cussed  in  detail. 
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Overview 

The  objective  of  this  chapter  is  to  thoroughly  explain 
and  justify  our  research  design  and  methodology  used  to 
answer  Investigative  Question  3.  A  brief  suiTimary  of  the 
problem  is  presented  followed  by  the  detailed  methodology, 
description,  and  justification  of  how  we  answered  this  spe¬ 
cific  investigative  question. 

Problem  Summary 

In  order  to  address  the  problem  of  revising  the  course 
curriculum  for  CON  331  -  Executive  Cost  and  Price  Analysis, 
the  following  investigative  questions  must  be  answered. 

1.  Who  IS  presently  required  to  take  the  oourse? 

2.  What  are  the  current  course  objectives  and  conrer..: 

3.  What  specific  topics  represena  the  needs  or  she 
people  who  are  required  ro  rake  the  course? 

Investigative  Questions  1  and  2  were  answered  in  Chapte 
II,  Literature  Review.  Specifically,  Question  1  (Who  is 
presently  required  to  take  the  course?)  was  answered  by 
reviewing  DOD  Manual  5000. 52-M,  "Career  Development  Program 
for  Acquisition  Personnel",  November  1992  and  Question  2 
(What  are  the  current  course  objectives  and  content?)  by 
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reviewing  the  present  course  syllabus  and  cur 
interviewing  the  course  director.  The  third 
question  will  be  answered  via  a  questionnaire 
using  the  Delphi  technique  as  described  below 


^  z.  c  A  ^  n  ci 
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Question  3  Methodology 

Question  3  (What  specific  topics  represeni  ihe  needs  of 
the  people  required  to  take  the  course?)  was  answered  using  a 
questionnaire  methodology  utilising  the  Delphi  technique. 

The  questionnaire  method  resulted  in  a  list  of  the  most 
needed  topics  for  CON  331  which,  in  turn,  served  as  the  basis 
of  the  revised  curriculum. 

Description  of  Delphi  --  When  to  Use  and  Current  p  p  1 1  - 
cation.  The  Delphi  technique  is  a  decision  making  aid  that 
elicits  a  consensus  judgment  or  opinion  from  a  panel  of 
experts  through  an  iterative  questionnaire  and  feedbacl: 
process.  The  method  is  most  useful  whenever  insufficient 
no  applicable  data  exits,  the  required  data  is  too  expensive 
to  obtain  and  analyse,  or  the  significant  problem  variables 
and  their  interaction  are  not  clearly  known  (Zaloom, 

1974:32).  A  more  detailed  description  of  the  Delphi  tech¬ 
nique  is  provided  in  Appendix  A. 

While  there  exists  an  abundance  of  data  concerning 
managerial  and  executive  training  in  general,  as  well  as 
numerous  studies  in  specific  areas,  e.g.,  training  needs  fcr 
accounting  executives  and  managers;  as  indicated  in  the 
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previous  chapter,  no  relevant  data  exists  concerning  specif¬ 
ic,  topical  educational  needs  for  pricing  executives  and 
managers.  The  most  appropriate  methodology  for  this  purpose 
is  deemed  to  be  the  Delphi  technique  (Clshfski  and  Joseph, 
1991:297).  The  Delphi  technique  requires  the  determination 
and  selection  of  experts. 

Requirements  of  Experts  and  Current  Panel  Sel ection . 

The  sise  and  composition  of  the  panel  of  experts,  as  well  as 
the  nature  of  the  questionnaires  and  method  of  tabulation, 
varies  among  areas  of  use.  The  experts  must  have  knowledge 
of  the  problem  area,  have  a  good  performance  record  in  their 
areas  of  expertise,  have  a  high  degree  of  objectivity  and 
rationality,  and  commit  the  time  required  for  the  study 
(Tersine  and  Riggs,  1976:53-54).  For  education  needs  assess¬ 
ment,  the  experts  should  also  be  broadly  representative  of 
the  population  that  is  being  assessed,  and  have  e::perien:r  i.. 
the  organisation  and  therefore  know  more  about  life  inside 
the  organisation  than  anyone  on  the  outside  (Clshfski  and 
Joseph,  1991:298). 

Since  the  problem  of  identifying  training  topic  require¬ 
ments  for  pricing  managers  and  executives  is  narrowly  de¬ 
fined,  a  relatively  homogeneous  panel  was  required.  There¬ 
fore,  this  study  employs  a  panel  of  six  experts.  The  experts 
were  chosen  based  on  each  experts  current  position,  our 
personal  judgements,  and  the  judgements  of  our  thesis  advi¬ 
sors.  Table  3.1  reflects  the  positions  held  and  organisa- 
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ticns  represented  by  the  experts.  (Tj  assure  anonymity,  the 
list  of  positions  and  organisations  are  not  presented  rn  a 
one-to-one  correspondence.) 

TABLE  3.1 

POSITIONS  AND  ORGANIZATIONS  OF  EXPERT  PANEL 


Position  Title 

Chief,  Price/Cost  and  Financial 
Analysis  Branch 
Chief,  Pricing  Division 
Director,  Pricing  and  Finance 
Chief,  Contract  Review  and 
Pricing  Division 
Staff  Price  Analyst 
Supervisory  Contract  Negotiator 


The  experts  are  all  DoD  civilian  employees  in  grades  GS/GM 
14-15  and  represent  a  combined  total  acquisition  experience 
base  in  excess  of  ICC  years. 

The  panelist  supervise,  or  establish  contract  pricing 
policy  for  these  whom  the  course  is  required.  Three  experts 
represent  staff  positions  and  three  should  represent  buying 
or  field  positions.  Price  analysts  in  staff  positions  are 
concerned  with  the  regulations  affecting  the  buying  positions 
and  with  problems  which  go  up  through  channels  to  the  staff 
for  action.  Buying  or  field  cost  and  price  analysts  support 
Contracting  Officers  by  performing  cost  analysis  and  leading 
in  price  negotiations  in  large  dollar  purchases  or  contract 
modifications  and  by  completing  other  related  pricing  tasks 
assigned  by  the  agency. 


ASC 

SM-AIC 

SAF 

OC_ALC 
HQ,DLA 
HQ,  AFMC 
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Two  of  the  experts  have  extensive  ocntract  administra¬ 
tion  experience.  Four  of  the  experts  have  extensive  purchas¬ 
ing  experience.  Two  of  the  experts  directly  represent  the 
customer  of  the  cost  and  price  analyst,  the  Contracting 
Officer.  Four  of  the  six  experts  are  in  positions  that 
require  them  to  take  the  course.  All  of  the  experts  are 
required  to  take  the  course  if  they  are  to  receive  their 
Level  III  certification  in  the  pricing  career  field. 

Based  upon  the  results  of  who  is  presently  required  ro 
take  this  course  and  discussions  with  our  thesis  advisors,  we 
believe  that  the  above  described  experts  constitutes  a  panel 
of  experts  qualified  to  recommend  topics  to  be  covered  i.a  a 
course  for  Level  III  certification  for  pricing  managers  and 
executives.  This  mix  of  experts  ensures  that  all  aspects  of 
pricing  management  are  represented,  that  the  population  of 
individuals  who  are  required  to  take  the  course  are  rep.--- 
sented,  and  that  there  is  a  high  degree  of  customer  i.-.put. 

Use  of  Delphi  in  Educational  Needs  Assessment  and 
Curriculum  Design .  Miles  Weaver  (1933:18),  and  Dorothy 
Olshfski  and  Alma  Joseph  (1991:297)  advocate  the  use  of  the 
Delphi  technique  in  training  need  assessment  and  curriculum 
design.  Both  listed  potential  topics  and  asked  the  panelists 
to  indicate  the  relative  need  of  each.  Weaver  used  a  Likert 
type  scale  and  used  a  simple  mean  of  the  scores  for  the 
individual  topics  to  establish  the  ranking  of  the  topics  from 
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most  important  to  least  important.  Olshfski  and  Joseph  asked 
the  panel  to  rank  the  topics  from  most  important  to  least 
important  and  then  summarized  the  responses  to  obtain  a 
composite  ranking.  The  summaries  were  sent  back  to  the 
experts  for  comment  and  reconsideration.  The  iterative 
process  continued  until  a  consensus  was  reached  concerning 
the  most  important  topics  to  be  covered  in  subsequent  trsi.n.- 
ing  courses.  Each  researcher  solicited  additional  topics 
beyond  those  listed  on  the  first  questionnaire  and  neither 
researcher  required  more  than  three  iterations  before  a 
consensus  was  obtained.  This  same  process  was  used  in  this 
research.  However,  a  prerequisite  was  the  need  for  a  list  o 
possible  topics.  This  was  obtained  through  a  seperate  survey 
of  CON  331  students. 

Survey  Design .  The  list  of  possible  topics  for  CCN 
331  to  be  used  in  the  initial  questionnaire  fcr  the  panel  of 
experts  was  established  as  described  below. 

A  survey  was  given  to  the  students  of  the  October  1992 
and  February  1993  CON  331  course  offerings.  Recall  that  this 
course  currently  provides  the  journeyman  price  analyst  with 
training  to  improve  their  technical  skills  and  increase  their 
technical  level  of  competency.  Also  note  that  these  classes 
were  not  completely  composed  of  only  pricing  managers  and 
executives.  The  topics  contained  in  the  surveys  were  deter¬ 
mined  by  the  researchers.  This  determination  was  based  on 
personal  experience  and  informal  discussions  with  several 
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individuals  in  the  pricing  career  field.  In  additicn  tc  the 
suggested  topics,  recommendations  were  solicited  from  the 
survey  participants  for  additional  topics.  We  surveyed  these 
students  to  search  for  potential  topics  to  use  in  the  Delphi 
technique  survey.  The  survey  instrument  and  the  results  are 
contained  in  Appendices  B  and  C  respectively. 

In  order  tc  ensure  a  wide  variety  cf  topics  fcr  the 
e.xperts  questi  onnai  r  o ,  comments  from  course  critiques  com¬ 
pleted  by  students  throughout  1932,  and,  for  comparison 
purposes,  critiques  f.cm  ISS”  were  reviewed,  as  well  as 
literature  regarding  the  topics  of  concern  for  cost  and  pr 
analysts  was  explored.  Based  on  this  information  and  the 
results  of  the  student  surveys  an  initial  questionnaire  to  be 
administered  to  the  e.'iperts  was  established. 

Initial  Questionnaire.  The  questionnaire,  developed  ro 
described  above,  was  sent  tc  the  panel  of  experts.  The 
experts  were  requested  to  rate  the  topics  based  on  the 
tance  of  the  topic  as  it  relates  to  their  needs.  The 
is  based  on  a  scale  of  1  to  5;  1  representing  the  lev.  ^-st 
level  of  importance  and  5  representing  the  highest  levs-1  of 
importance.  In  addition,  we  requested  recommendations  from 
the  experts  relating  to  topics  that  were  not  specifically 
listed  in  the  questionnaire.  The  initial  questionnaire  is 
contained  in  Appendix  D. 
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Tabulation  of  Results  and  Formul ation  of  Second  Ques¬ 
tionnaire.  Cnee  the  results  were  received,  the  raw  data  was 
combined  into  a  frequency  of  rating  for  each  topic.  Since 
the  objective  of  this  questionnaire  was  to  determine  the  most 
important  topics,  we  grouped  the  data  into  two  categories; 
category  A  representing  topics  where  at  least  3  of  the  6 
e.xperts  initially  weighted  a  topic  4  or  5  (high  on  the  scale 
of  importance),  representing  a  consensus  of  importance,  and 
category  3  representing  the  remainder  of  the  topics.  The 
remainder  of  the  questionnaire  process  concerned  itself  with 
only  the  category  A  topics  (as  they  represent  the  group 
consensus  as  to  the  important  topics)  and  all  other  topics 
initially  recommended  by  the  experts  (in  order  to  give  the 
entire  group  of  experts  a  chance  to  weight  these  topics). 

Second  Questionnaire  Process .  The  second  questionnaire 
consisted  of  a  listing  of  all  of  the  topics  that  the  experts 
determined  to  be  important  (category  A  topics)  along  with  thi 
recommendations  submitted  in  the  initial  survey.  In  this 
questionnaire,  the  experts  were  requested  to  ran.’-;  the  topics 
in  order  of  most  important  to  least  important.  In  addition, 
the  experts  were  asked  to  provide  specific  areas  of  need 
(subtopics)  relating  to  at  least  their  five  most  important 
general  topics.  The  second  questionnaire  is  contained  in 
Appendix  F. 

Tabulation  of  Resul ts .  The  results  of  the  second  ques¬ 
tionnaire  were  tabulated  by  topic.  We  then  performed  Ken- 
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iall'3  -osffijient  of  concordance  test  to  deterTiine  the 
degree  of  agreement  among  the  experts  for  the  rankrr.g  of  :he 
topics.  (An  explanation  of  how  Kendall's  coeffici>=nt  of 
concordance  works  is  contained  in  Appendix  H.)  Having  estab¬ 
lished  a  minimum  confidence  level  of  85%  and  since  the  test 
indicated  agreement  between  the  experts  at  a  significance 
level  of  less  than  15%  when  adjustments  were  made  for  the 
pclariaed  topics,  only  two  iterations  of  the  questionnaire 
were  required. 

Survey  Instrument  Validity .  The  topic  selection  proce¬ 
dure  described  above  addresses  the  content  validity  cf  the 
instrument.  Concurrent  and  construct  validity  are  estab¬ 
lished  via  two  approaches.  The  first,  which  is  more  con¬ 
cerned  with  forecasting,  compares  the  Delphi  predictions  -.vioh 
actual  occurrences.  Experimental  results  show  that  Delphr  is 
"at  least  a  'good'  predictor  with  predictions  relatively 
close  to  actuality"  (Tersine  and  Riggs,  1571:56).  Iho  s;..,..  1 
compares  Delphi  with  other,  more  conventional  methods  on-1 
here,  too,  the  Delphi  method  has  been  found  to  be  a  moie 
accurate  forecaster  (Tersine  and  Riggs,  1976:55).  Regarding 
the  validity  of  using  the  Delphi  technique  for  developing 
curriculum  requirements.  Weaver  states  "The  results  you 
obtain  in  a  Delphi  study  represent  a  firm,  locally  valid 
basis  on  which  you  can  build  a  relevant  curriculum"  (Weaver, 
1988:20).  The  "local"  nature  of  this  study,  relating  to  its 
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validity,  rests  in  the  very  narrow  topic  cf  interest,  tra 
ing  for  pricing  executives  and  managers. 

Summary 

Previously,  the  question  hierarchy  and  the  pertinent 
literature  regarding  the  investigative  questions  were  re¬ 
viewed.  This  chapter  thoroughly  explained  and  justified 
research  design  and  methodology.  A  questionnaire  methcdo 
based  on  the  Delphi  technique  was  developed  to  establish 
list  of  suggested  course  topics  and  subtopics  to  be  used 
the  CON  331  course  curriculum.  The  next  chapter  provides 
insight  into  the  data  description  and  analysis  of  the  re¬ 
search. 
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Chapter  IV 

Findings  and  Analysis 


Introduction 

This  chapter  presents  the  results  of  our  research  effort 
as  described  in  Chapter  III.  The  results  and  data  analysis 
fcr  the  student  surveys  (used  tc  develop  rhe  experts  ques¬ 
tionnaire)  and  the  two  iterations  cf  expert  questionnaires 
are  be  presented  followed  by  the  specific  findings  as  they 
relate  to  our  investigative  questions. 

Student  Surveys 

As  described  in  Chapter  III,  the  purpose  cf  the  student 
surveys  was  to  gather  input  data  for  the  f orrr.ulation  of  th= 
expert's  questionnaire.  The  Cctober  1392  CC'I  211  rlise  '..•us 
surveyed  to  assist  in  de terraining  topics  for  the  e.xpert 
questionnaire.  The  February  1993  CCH  331  ciass  was  also 
surveyed  to  validate  the  initial  survey  results  and  search 
fcr  additional  expert  questionnaire  copies.  The  survey  green 
to  both  classes  is  contained  in  Appendix  B. 

The  general  format  of  the  surveys  is  as  follows.  The 
survey  begins  with  an  introductory  paragraph  explaining  the 
purpose  of  the  survey.  Next,  there  is  a  request  for  demo¬ 
graphic  information  (i.e.  current  position,  length  in  current 
position,  grade  or  rank,  and  years  experience  at  given  lev- 
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els).  This  information  was  necessary  in  order  to  have  a 
general  idea  of  the  backgrounds  of  ths  survey  respondents.  A 
listing  of  possible  topics  was  provided  and  the  respondents 
were  asked  to  rate  them  in  order  of  need  for  their  current 
position  with  1  representing  a  low  need  and  5  representing  a 
high  need.  This  was  followed  by  a  request  for  any  additional 
topics  not  contained  in  the  survey. 

Demographic  Data  Summary .  Summarized  in  Table  4.1  is 
the  demographic  data  collected  during  the  student  surveys. 

The  demographic  data  includes  the  total  number  of  survey 
respondents,  whether  they  hold  supervisory  or  nonsupervisory 
positions,  their  grade  or  rank,  and  the  average  pricing 
experience  of  the  group. 


TABLE  4.1 

DEMOGRAPHIC  DATA  OP  STUDENTS  SURVEYED 


Total  survey  respondents;  27 

Supervisory /Nonsupervisory  11/15 

Grade/Rank : 

G3/GM-14:  3  GS/CM-13:  11  GS-12:  3 

Lt  Ccl :  2  Maj:  1  Capt :  1 

Average  Pricing  Experience;  8.0  years 


Survey  Resul ts .  Table  4.2  lists  the  results  from  the 
student  surveys  presented  in  ranked  order  of  average  weight¬ 
ing  (The  individual  raw  data  rankings  are  contained  in  Appen¬ 
dix  C.).  The  table  lists  each  specific  topic  and  the  average 
rating  the  topic  received  in  the  student  survey. 
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TABLE  4.2 


RESULTS  OF  STUDENT  SURVEYS 


Average 

Topic  Ratine 

Estimates  vs  Actuals  4.2 

Incentive  Arrangements  4.2 

Estimating  4.1 

Evaluation  of  Bill  of  Material  4.1 

Defective  Pricing  4.1 

*  Analysis  of  Technical  Input  3.9 

*  Analysis  of  DCAA  Input  3.8 

Claims  3.7 

Overhead  (Unallowables)  3.6 

*  Source  Selection  3.6 

Statistical  Tools  3.6 

Requests  for  Equitable  Adjustments  3.5 

*  Factfinding  3.5 

Cost  Accounting  Standards  3.5 

Lease  vs  Buy  3.5 

Disputed  Claims  3.4 

Uncompensated  Overtime  3.4 

Termination  for  Convenience  3.3 

Ethical  Negotiations  3.2 

Warranties  3.2 

Acquisition  Professional 

Development  Program  3.1 

Termination  for  Default  3.1 

Laws ,  Cases  3 . 1 

Depreciation  ;.C 

Value  Engineering  Change  Proposal  2.9 

Impact/Cost  of  CPR,  Mil  Std  1567A  2.9 

Management  Information  Systems  2.7 

Alternate  Disputes  Resolution  2.7 

Subordinate  Training  2.6 

Ethical  Conduct  2.5 

Deferred  Compensation  2.6 

Contract  Basics  2.5 

*  Foreign  Military  Sales  Pricing 

Support  2.4 

Court  Appearances  2.1 

Voluntary  Refunds  2.1 

Defense  Base  Operating  Fund  2.0 

*  Depot  Competition  1.9 

*  Topics  suggested  by  students 
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Formulation  of  Expert  Questionnaire .  As  explained  in 


Chapter  III,  the  purpose  of  the  student  surveys  was  to  deter¬ 
mine  topics  to  include  in  the  expert  questionnaire.  In  order 
to  have  a  wide  variety  of  topics  in  the  first  expert  ques¬ 
tionnaire,  we  decided  not  to  eliminate  any  possible  topics. 
Therefore,  all  of  the  topics  contained  in  the  students  survey 
were  included  in  the  first  expert  questionnaire. 

First  Expert  Questionnaire 

As  previously  stated  in  Chapter  III,  the  purpose  of  the 
first  expert  questionnaire  was  to  present  a  list  of  possible 
topics  for  the  experts  to  rate  in  order  of  preference  for  use 
in  developing  the  curriculum  for  CQN  331.  The  experts  were 
requested  to  score  all  of  the  topics  on  a  scale  of  1  to  5 
with  1  representing  a  low  need  and  5  representing  a  high 
need.  In  addition  to  scoring  the  topics,  the  experts  were 
also  asked  to  include  any  topics  that  they  considered  impor¬ 
tant  but  were  not  included  in  the  questionnaire. 

Questionnaire  Resul ts .  Illustrated  in  table  4.3  are  th, 
results  of  the  first  expert  questionnaire.  The  topics  are 
listed  as  they  appeared  on  the  questionnaire.  The  table 
lists  each  topic  and  its  frequency  of  rating.  The  frequency 
of  rating  represents  how  the  group  of  experts  rated  a  partic¬ 
ular  topic.  For  instance,  ethical  negotiations  received  1 
rating  of  3,  1  rating  of  4,  and  4  ratings  of  5.  The  raw  data 
scores  from  this  questionnaire  are  contained  in  Appendix  E. 
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TABLE  4.3 


FREQUENCY  OF  SCORES  OF  TOPICS  RESULTING  FROM 
FIRST  EXPERT  RESPONSE 


Topics 

1 

Frequency 
2  3  4 

5 

Ethical  Negotiations 

0 

0 

1 

1 

4 

Laws,  Cases 

0 

0 

1 

2 

3 

Ethical  Conduct 

0 

0 

1 

2 

3 

Overhead  (Unallowable) 

1 

0 

1 

2 

2 

Estimating 

1 

1 

0 

2 

2 

Depot  Competition 

2 

0 

0 

2 

X 

Contract  Basics 

0 

1 

1 

dm 

Incentive  Arrangements 

0 

0 

0 

5 

X 

Cost  Performance  Reports, 
Work  Measurement,  etc. 

0 

1 

1 

3 

1 

Subordinate  Training 

Q 

0 

0 

6 

0 

Factfinding 

1 

1 

X 

X 

0 

3 

Cost  Accounting  Standards 

1 

1 

1 

X 

1 

Estimates  vs  Actuals 

]_ 

0 

0 

dm 

2 

1 

Request  for  Equitable 
Adjustment 

X 

X 

2 

X 

Source  Selection 

0 

0 

3 

c 

Claims 

1 

1 

1 

Cf 

0 

Alternate  Disputes 

Resolution 

1 

1 

3 

c 

X 

Defective  Pricing 

1 

0 

3 

1 

1 

Foreign  Military  Sales 
Pricing  Support 

0 

3 

1 

1 

Evaluation  cf  Bill  of 
Materials 

1 

1 

4 

r\ 

\j 

0 

Statistical  Tools 

c 

X 

X 

X 

0 

Court  Appearances 

1 

1 

u 

Depreciation 

1 

1 

1 

O 

V 

Warranties 

o 

0 

4 

C 

0 

Disputed  Claims 

1 

X 

1 

C 

Management  Information 
Systems 

0 

1 

4 

1 

0 

Deferred  Compensation 

2 

1 

1 

2 

0 

Uncompensated  Overtime 

3 

1 

0 

1 

1 

Termination  for  Default 

2 

2 

2 

0 

A 

Voluntary  Refunds 

2 

2 

0 

2 

0 

Defense  Base  Operating 

Fund 

3 

1 

1 

1 

0 

Value  Engineering  Change 
Proposal 

2 

2 

2 

0 

0 

Termination  for  Convenience 

2 

2 

2 

0 

0 

Lease  vs  Buy  Decisions 

1 

3 

1 

1 

0 
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Anal  vs is  Performed .  As  explained  in  Chapter  III 


the 


topics  were  divided  into  two  groups.  Group  A  and  Group  B. 
Group  A  represented  the  topics  in  which  at  least  3  of  the  6 
experts  scored  a  given  topic  as  a  4  or  5 .  This  represented  a 
consensus  as  to  an  important  topic.  The  remainder  of  the 
topics  were  placed  in  Group  B.  The  Group  B  topics  were 
excluded  from  the  remainder  of  the  research  as  they  represent 
topics  in  which  expert  interest  as  a  group  was  not  high. 

Shown  in  Tables  4.4  and  4.5  are  the  group  A  and  B  topics 
resulting  from  this  analysis.  Table  4.4  represents  the 
topics  which  were  included  in  the  second  experts  question¬ 
naire.  Table  4.5  reflects  those  topics  which  the  experts  did 
net  reach  a  consensus  as  to  being  important  topics  i_.  cc.'.  ■ 
sideration.  Both  lists  of  topics  are  listed  in  order  of 
their  mean  average  score,  from  highest  to  lowest. 

The  topics  shown  in  Table  4.5  were  suggested  by  the 
e.'iperts  for  inclusion  as  additional  topics  in  the  ne.xt  iuoo  - 
tionnaire.  This  list  represents  recommendations  frc;v.  the 
experts  as  to  topics  they  individually  believe  to  be  of 
importance  to  pricing  executives  and  managers. 

Formulation  of  Second  Expert  Questionnaire .  The  second 
expert  questionnaire  consisted  of  the  group  A  topics  from  the 
first  questionnaire  that  the  experts  agreed  were  important 
(shown  in  Table  4.4)  and  any  topic  that  was  recommended  by  an 
individual  expert  (shown  in  Table  4.6).  This  was  necessary 


4-6 


TABLE  4.4 


TOPICS  TO  BE  INCLUDED  IN  THE  SECOND  QUESTIONNAIRE  (GROUP  A) 


Ethical  Negotiations 
Laws,  Cases 
Ethical  Conduct 
Overhead  (Unallowable) 

Estimating 
Depot  Competition 
Contract  Basics 
Incentive  Arrangements 

Cost  Performance  Reports,  Work  Measurement 

Subordinate  Training 

Factfinding 

Cost  Accounting  Standards 

Estimates  vs  Actuals 

Request  for  Equitable  Adjustment 

Source  Selection 

Claims 


TABLE  4.5 

TOPICS  TO  BE  EXCLUDED  FROM  SECOND  QUESTIONNAIRE  (CRCUP  E) 


Alternate  Disputes  Resolution 
Defective  Pricing 

Foreign  Military  Sales  Pricing  Support 

Evaluation  of  Bill  of  Materials 

Statistical  Tools 

Court  Appearances 

Depreciation 

Warranties 

Disputed  Claims 

Management  Information  Systems 
Deferred  Compensation 
Uncompensated  Overtime 
Termination  for  Default 
Voluntary  Refunds 
Defense  Base  Operating  Fund 
Value  Engineering  Change  Proposal 
Termination  for  Convenience 
Lease  vs  Buy 
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TABLE  4  .  5 


ADDITIONAL  TOPICS  SUGGESTED  BY  EXPERT  PANEL 


Adequate  Price  Competition  for  Other  than  Firm-Fixed-Price 

Pricing  Arrangements 

Contractor  Financing 

Cost  Monitoring  Reviews 

DoD  Profit  Policy 

Eichleay  Formula  for  Unabsorbed  Overhead 
Pricing  of  Commercial  Items 

Program  Management/Pricing  Conflict  Resolution 
Roles/Responsibilities  of  a  Price  Analyst  in  Negotiations 
Specialized  Cost  Reviews  ie.  Should  Cost 


in  order  to  allow  the  entire  panel  of  experts  to  rank  the 
individually  suggested  topics. 

Second  Expert  Questionnaire 

The  second  expert  questionnaire  requested  the  e.xperts  :  - 
rank  all  of  the  topics  in  order  of  importance  from  1  to  25  ( 1 
representing  the  .most  i.mportant  topic  and  25  representing  the 
least  important).  In  addition,  each  expert  was  requested  to 
provide  comments  relating  ro  at  least  their  top  5  topics. 

The  purpose  of  the  comments  was  to  receive  suggested  areas  cf 
discussion  relating  to  individual  topics.  The  goal  of  this 
questionnaire  was  to  establish  a  prioritized  list  of  suggest¬ 
ed  topics  and  comments  for  at  least  the  top  10  topics. 

Questionnaire  Resul ts .  Shown  in  Table  4.7  is  the  list 
of  topics  and  the  rankings  that  each  individual  expert  as¬ 
signed  each  topic.  The  table  lists  each  topic  and  indicates 
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the  ranking  that  each  expert  gave  the  topic.  The  topice  are 
listed  in  the  order  that  they  were  presented  in  the  question¬ 
naire  . 

TABLE  4.7 

RANKINGS  OF  INDIVIDUAL  TOPICS  BY  EXPERT 


Topic 

Expert 

3  4 

5 

r 

Ethical  Negotiations 

18 

24 

16 

1 

Current  Laws  and  Cases 

9 

20 

1  '5 

4 

Ethical  Conduct 

19 

4 

6 

Incentive  Arrangements 

7 

10 

G 

4 

Establishing  A  Training  Program 
for  Subordinates 

13 

5 

10 

15 

5 

Fundamentals  of  Contract  Theory 

3 

24 

15 

5 

0 

2  4 

Overhead  (Unallowables) 

14 

C 

21 

7 

Impact/Cost  of  CPR, 

Mil  Std  1567A,  etc 

13 

18 

1  ♦ 

^  1 

20 

Source  Selection 

1  Z 

4 

17 

1  ^ 

Estimating 

11 

19 

12 

T  r\ 

X  ^ 

Factfinding 

12 

20 

4. 

1  1 

Estimates  vs  Actuals 

10 

21 

4 

•  r 

- 

Depot  Competition 

1 

14 

14 

.. 

Cost  Accounting  Standards 

15 

2  3 

6 

1  1 

1  G 

Requests  for  Equitable 

Adjustment 

4 

-  - 

^  0 

Claims 

■X 

17 

c 

" 

Adequate  Price  Competition  for 
Other  than  FFP 

3 

11 

Q 

o 

17 

1  ? 

X  w 

Contractor  Financing 

• 

r 

20 

i  2 

; 

Cost  Monitoring  Reviews 

24 

25 

13 

^  z 

w  - 

DoD  Profit  Policy 

20 

8 

G 

^  2. 

14 

Eichleay  Formula  for 

Unabsorbed  Overhead 

5 

16 

21 

24 

’  - 

Pricing  of  Commercial  and  Non- 
development  a  1  Items 

16 

6 

11 

9 

23 

19 

Program  Management/Pricing 
Conflict  Resolution 

22 

1 

19 

24 

IS 

T 

X 

Roles/Responsibilities  of  the 
Price  Analyst  in  Negotiations 

17 

7 

3 

25 

9 

0 

Specialized  Cost  Reviews 

23 

9 

12 

10 

S 

18 

iO 


Table  4.8  lists  all  of  the  comments  provided  by  the 
experts  for  their  five  highest  individually  ranked  topics 
The  table  lists  the  comments  from  the  experts  relating  to 
specific  recommendations  on  areas  to  be  discussed  under 
individual  topics.  The  co.mments  are  listed  in  the  order 
received. 


TABLE  4.8 

COMMENTS  ON  SPECIFi:  TOPICS 


Comments : 

Program  Management/Pricing  Conflict  Resolution 
-Program  management's  mission 
-Pricing's  mission 
-Resolution  of  the  conflict 

Roles/Responsibilities  of  the  Analyst  in  Negotiatic..s 
-Role  of  Contracting  Officer 
-Role  of  the  Price  Analyst* 

-Role  of  the  field  agencies 
-Role  of  other  technical  support 

Ethical  Settlements  in  Negotiations 
-What  is  a  settlement? 

-What  ethics  are  involved  in  negotiations? 

-Do  laws  protect  the  Government  from  poor  sth*os 
the  contractor's  part? 

-Equity  for  both  Government  and  contractor 

Current  Laws,  Court  Cases,  Regulations 
-FAR,  DFARS 
-ASBCA  cases 

Ethical  Conduct 

-Combine  with  "Ethical  Negotiations" 

-Truth  in  Negotiations  Act 
-Disclosure  of  unallowable  costs 

Incentive  Arrangements 
-Types  of  contracts 
-Performance  incentives 
-Tradeoffs 
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TABLE  4.8  (CONTINUED) 


COMMENTS  ON  SPECIFIC  TOPICS 


Establishing  a  Training  Program  for  Subordinates 
-Need  for  technical  training  in  such  areas  as 
price/cost  analysis,  cost  estimating,  contract  admin¬ 
istration,  cost  accounting  standards,  overhead  analy¬ 
sis,  etc. 

Factfinding 

-What  to  look  for 

-How  to  conduct  meetings 

Requests  for  Equitable  Adjustments 
-Rules  for  establishing  price 

-Contract  terms,  e.g.  Not-To-Exceed,  Changes  Clause 
-Differences  between  equitable  adjustment  and  repricing 
-How  to  achieve  "leave  them  as  you  found  them"  while 
complying  with  TIN.A 

-Form  in  which  proposal  should  be  submitted 

Source  Selection 

-Methods  of  evaluation 
-Rul es 

-How  to  perform  cost  realism  assessment  with  limited, 
uncertified  cost/price  data 
-Define  realistic,  complete,  and  reasonable 
-How  to  handle  uncorrected  CR’s 

Fundamentals  of  Contract  Theory 
-Contract  provisions  vs  price 
-Laws 

-Contract  types 

Depot  Competition 

-Relationship  between  Cost  Comparabi  1  i ty  Ha.idb.,ok  and 
Cost  Accounting  Standards 

-How  to  perfoim  meaningful  comparisons  of  public  and 
private  proposals 

-Overview  of  differences  in  costing  practices  among 
the  services 

Adequate  Price  Competition  for  Other  than  FFP 

-How  to  structure  evaluation  criteria  to  obtain  apples 
to  apples  comparison 
-Examples  of  contract  arrangements 
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TABLE  4.8  (CONTINUED) 


COMMENTS  ON  SPECIFIC  TOPICS 

Eichleay  Formula  for  Unabsorbed  Overhead 
-Case  law  history 

-Premise  upon  which  entitlement  is  established 
-Formula  for  calculation 
-Burden  of  proof 

-.Al  1  owabi lity/unal  1  owabi  1  ity  of  variable  and  semi¬ 
variable  costs  in  overhead  pool 


Analysis  Performed .  A  composite  ranking  of  the  experts 
individual  rankings  was  developed  by  summing  the  rank  as¬ 
signed  by  each  of  the  experts  to  each  tcpic.  The  composite 
ranking  was  then  prioritized  from  most  preferred  to  least 
preferred.  A  calculated  Kendall’s  coefficient  of  conccrdanc 
(W)  of  .183,  with  the  associated  Chi-squared  value  of  26.4 
and  p-value  of  .334,  indicates  the  degree  of  consensus  is  no 
very  high  (Kendall:  Chapter  3).  (Reference  Appendix  K  for  a 
more  detailed  discussion  of  Kendall's  coefficient  of  concord 
ance  (W),  related  statistical  measures,  and  specific  formu¬ 
lae.)  The  resulting  prioritised  composite  ranking  is  she:-;.-, 
in  Table  4.9,  as  is  the  analysis  of  the  data. 

A  "polarized"  topic  is  defined  to  exist  whenever  at 
least  two  respondents  ranked  the  tcpic  in  the  first  quartile 
and  at  least  two  ranked  it  in  the  last  quartile.  This  polar 
ization  of  opinions  on  some  issues  is  expected  and  reflects 
valid  observations  or  needs  based  upon  the  respondents  loca¬ 
tions  and/or  experiences. 
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TABLE  4 . 9 


ANALYSIS  OF  RESPONSES  TO  SECOND  QUESTIONNAIRE 


ITEM  CXPEST  i: 

Incantiv*  Arranaeamta 

Currant  Lawa  and  Casas 

Ethical  Conduct 

Sourca  Salaction 

Establishinu  a  Traininu  Prograa  for 
Suhordinatas 

Rolaa/Raaponsibilities  of  the  Pric«  analyst 
in  Na^otiations 
Ethical  Haaotiations 
Factfindinu 

Maquata  Prica  Conpatition  for  Other  than 
Fin  Fixed  Prica 
Depot  Coapatition 
Requests  for  Equitable  Adjustawnts 
Estiaatinq 
Claias 

Overhead  (Unallowablea) 

Specialized  Cost  Reviews 
Fundaaantals  of  Contract  Thaorr 
Cost  Accountinq  Standards 
Pricing  of  Coaaarcial  and  Hondavalopaantal 
Itaas 

Prograa  Manageaant/Pricing  Conflict 
Resolution 
Estiaatas 
OoO  Profit  Policy 
Contractor  Financing 

Eichlaay  Foraula  for  Unabsorbad  Overhead 
lapact/Cost  of  CPR,  Mil  Std  1376A,  etc 
Cost  Monitoring  Reviews 


1 

2 

3 

4 

5 

6 

SUN 

(SUH-HS) 

7 

10 

9 

1 

4 

9 

40 

1444 

9 

2 

20 

13 

2 

4 

50 

784 

19 

4 

23 

6 

3 

5 

60 

324 

2 

15 

7 

4 

17 

15 

60 

324 

13 

5 

10 

15 

5 

14 

62 

256 

17 

7 

3 

25 

9 

2 

63 

225 

18 

3 

24 

16 

1 

3 

65 

169 

12 

20 

2 

2 

11 

20 

67 

121 

3 

11 

8 

17 

19 

10 

68 

100 

1 

14 

14 

7 

25 

11 

72 

36 

4 

22 

16 

3 

22 

6 

73 

25 

11 

19 

1 

12 

10 

21 

74 

16 

6 

17 

17 

8 

21 

7 

76 

4 

14 

18 

5 

21 

7 

13 

78 

0 

23 

9 

12 

10 

8 

IS 

80 

4 

8 

24 

15 

5 

6 

24 

82 

16 

15 

23 

6 

11 

16 

12 

83 

25 

16 

6 

11 

9 

23 

19 

84 

36 

22 

1 

19 

24 

18 

1 

85 

49 

10 

21 

4 

22 

15 

22 

94 

256 

20 

8 

22 

19 

14 

16 

99 

441 

25 

12 

25 

20 

12 

8 

102 

576 

5 

16 

21 

23 

24 

17 

106 

784 

21 

13 

18 

14 

20 

23 

109 

961 

24 

25 

13 

18 

13 

25 

118 

1600 

Total  sua  of  ranks 

1950 

Naan  of  sua  of  ranks  (MS) 

78 

Sua  of  squares  of  deviation 

froa  aean  (S) 

8576 

Coefficient  of  cortcordance 

(W) 

0.183 

Chi-squared  (X^) 

26.39 

Degrees  of  freedoa  (v) 

24 

p-value 

0.334 
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When  adjustments  are  made  for  the  four  "poiarised" 
topics,  the  resultant  coefficient  of  concordance  is  .230,  the 
associated  Chi-squared  value  is  27.5,  and  the  p-value 
is  .121,  meeting  our  established  significance  level  for 
determining  whether  a  consensus  exists. 

This  polarization  does  not  affect  the  overall  ranking  o 
the  topics,  but  does  affect  the  apparent  strength  of  the 
agreement  or  degree  of  consensus.  Without  adjusting  for  the 
polarized  topics,  the  data  reflects  a  57%  level  of  confidence 
that  the  ranking  reflects  a  consensus.  After  adjusting  for 
four  polarized  issues,  the  level  of  confidence  rises  to  38%, 
which  is  considered  appropriate  for  decisions  regarding 
issues  as  general  as  education.  Table  4.10  presents  the 
analysis  of  the  data  with  the  adjustments  for  the  polarized 
topics.  The  polarized  topics  are  indicated  with  an  asterisk. 

Findings 

Investigative  Question  1.  Investigative  question  1,  Who 
is  presently  required  to  take  the  course?,  was  answered  in 
Chapter  2,  Literature  Review.  We  found  that  any  individual 
in  the  contracting  career  field  (GS-1102  and  comparable 
military)  who  has  the  grade  of  GS/GM-13  or  above  the  or  rank 
of  0-4  or  above  and  holds  a  job  that  is  "primarily  cost  and 
price  analysis  oriented"  is  required  to  have  completed  CON 
331  --  "Executive  Cost  and  Price  Analysis"  (DOD  1991;  B.1-5). 
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TABLE  4.10 


ADJUSTED  ANALYSIS  OF  RESPONSES  TO  SECOND  QUESTIONNAIRE 


ITEM  EXPERT  f: 

IncMitiv*  Arran9«Mnts 

Currant  L«wa  and  Caaai 

Ethical  Conduct 

Source  Sal action 

Estahlishina  a  Training  Prograa  for 
Subordinataa 

Rolaa/Raaponsibilitiea  of  the  Price  Anal rat 
in  Naaotiationa 
Ethical  Raaotiationa 
Pactfindina  * 

Adequate  Price  Conpetition  for  Other  than 
Fira  Fixed  Price 
Depot  Coapetition 

Requeata  for  Equitable  Adjuataents  * 

Eatiaatina 

Claiaa 

Overhead  (Unallowables) 

Specialized  Coat  Reviawa 
Fundaaentala  of  Contract  Theorr  * 

Coat  Accountina  Standarda 

Pricina  of  Coaaercial  and  Nondevelopaental 

Proaraa  Manageaent/Pricina  Conflict 
Reaolution  * 

Eatiaatea 
DoO  Profit  Policy 
Contractor  Financing 
Eichleay  Foraula  for  Unabaorbed  Overhead 
iBpact/Coat  of  CPR.  Hil  Std  1967A.  etc 
Coat  Monitoring  Reviews 


1 

2 

3 

4 

S 

6 

SUM 

(SUH-HS) 

6 

9 

8 

1 

4 

7 

35 

961 

7 

1 

16 

10 

2 

3 

39 

729 

16 

3 

19 

3 

3 

4 

48 

324 

2 

14 

6 

2 

IS 

13 

52 

196 

10 

4 

9 

12 

5 

12 

52 

196 

14 

6 

2 

21 

8 

1 

52 

196 

IS 

2 

20 

13 

1 

2 

53 

169 

3 

10 

7 

14 

16 

8 

58 

64 

1 

13 

13 

4 

21 

9 

61 

25 

9 

18 

1 

9 

9 

18 

64 

4 

5 

16 

14 

S 

18 

5 

63 

9 

11 

17 

4 

18 

6 

11 

67 

1 

19 

8 

11 

7 

7 

16 

68* 

4 

12 

20 

5 

8 

14 

10 

69 

9 

13 

5 

10 

6 

19 

17 

70 

16 

8 

19 

3 

19 

13 

19 

81 

225 

17 

7 

18 

16 

12 

14 

84 

324 

21 

11 

21 

17 

10 

6 

86 

400 

4 

15 

17 

20 

20 

15 

91 

625 

18 

12 

15 

11 

17 

20 

93 

729 

20 

21 

12 

15 

11 

21 

100 

1156 

Total  sua  of  ranks 

1386 

Mean  of  sua  of  ranks  (MS) 

66 

Sua  of  squares  of  deviation 

froa  aean  (S) 

6362 

Coefficient  of  concordance 

iV) 

0.230 

Chi-squared  (X^) 

27.54 

Degrees  of  freedoa  (v) 

20 

p-value 

0.121 
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Investigative  Question  2.  Investigative  question  2, 

What  are  the  current  course  objectives  and  content?,  was 
answered  in  Chapter  2,  Literature  Review.  We  found  the 
current  course  objective  was  to  address  specific  problems 
being  experienced  in  the  pricing  career  field  by  journeyman 
working  analysts  (Daneman,  1993).  The  current  course  de¬ 
scription  states  that  the  course  "Provides  experienced  pro¬ 
curement  analysts  with  statistic  and  accounting  tools  neces¬ 
sary  to  analyse  costs,  establish  cost  estimating  relation¬ 
ships,  and  develop  reasonable  cost  estimates  in  large  system 
procurements"  (AFIT  1989:  201-202).  Topics  in  the  current 
course  include  the  impact  of  unallowable  pool  and  base  costs 
on  overhead  allocation,  probability  and  dec-ision  analysis, 
tracking  actual  costs  to  estimated  costs  to  ascertain  bias, 
and  risk  analysis  and  production  quantity. 

Investigative  Question  3.  Investigative  question  3, 

What  specific  topics  represent  the  needs  of  the  people  who 
are  required  tc  take  the  course?,  was  answered  in  this  chap¬ 
ter.  By  questioning  a  panel  of  six  e.xperts  using  a  Delphi 
technique  as  explained  in  Chapter  3,  Methodology,  a  consensus 
was  reached  on  a  prioritized  list  of  topics.  '  In  addition  to 
the  list  of  topics,  comments  relating  to  specific  recommenda¬ 
tions  for  areas  of  study  for  an  individual  topic  were  re¬ 
ceived  for  fourteen  of  the  topics.  Illustrated  in  Table  4.11 
is  the  prioritized  list  of  recommended  topics  for  CQN  331  -- 
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Executive  Cost  and  Price  Analysis.  Thj  jGttui.en:s  relating  to 
the  specific  topics  are  located  in  Table  4.3. 

TABLE  4.11 

PRIORITIZED  LIST  OF  RECOMMENDED  TOPICS 


1.  Incentive  Arrangements 

2.  Current  and  Pending  Laws,  Court  Cases, 

Regul ations 

3.  Ethical  Conduct 

4.  Source  Selection  Pricing  Support 

5.  Establishing  a  Training  Program  for 

Subordinates 

o.  Roles  and  Responsibilities  of  Pricers  in 
Negotiations 

7.  Ethical  Settlements  in  Negotiations 

8.  Factfinding 

9.  Adequate  Price  Competition  for  Other  than  ?F? 

Contracts 

10.  Depot  Competition 

11.  Requests  for  Equitable  Adjustments 

12.  Estimating 

13.  Claims 

14.  Overhead  (Unallowables) 

15.  Specialized  Reviews,  e.g..  Should  Cost 

16.  Fundamentals  of  Contract  Theory 

17.  Cost  Accounting  Standards 

18.  Pricing  of  Commercial,  Ncndevei cpmentai  items 
IS.  Program  Management/Pricing  Conflict  ?.  e.:  c  1  ut  i  . 

20.  Estimates  vs  Actuals 

21.  Department  of  Defense  Profit  Policy 

22.  Contractor  Financing 

23.  Eichlea'y  Formula  for  Unabsorbed  Overhea 

24.  Impact  and  Cost  of  C/SCSC,  CPR ,  C/SSR, 

25.  Cost  Monitoring  Reviews 


The  following  is  a  summary  of  the  comments  provided  by 
the  panel  of  experts  for  the  top  ten  topics. 

Topic  1 .  The  points  of  emphasis  or  subtopics 
recommended  for  the  discussion  of  incentive  arrangements 
included  reviews  of  cost  and  fixed  price  incentive  contract.^ 
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performance  incentives,  working  with  multiple  incentives,  and 
possible  trade-offs. 

Topic  2 .  A  review  of  current  and  pending  laws, 
regulations,  and  board  or  court  cases,  including  changes  in 
the  Federal  Acquisition  Regulations  and  Department  of  Defense 
Federal  Acquisition  Regulation  Supplement,  should  emphasise 
new  trends  and  developments  as  opposed  to  administrative 
changes . 

Topic  3  (also  *7 ) .  The  comments  regarding  ethical 
conduct  were  very  closely  related  to  comments  relating  to 
ethical  settlements  in  negotiations.  Accordingly,  the  sub- 
topics  and  points  cf  emphasis  will  be  listed  together.  No 
comments  referenced  Air  Force  Regulation  30-30  or  similar 
ethics  regulations  or  credos.  The  subtopics  include  concepts 
such  as  fair  and  reasonable  settlements  and  open  ccmmunico- 
tions.  Specifically,  the  listed  subtopics  were  equity  for 
both  government  and  ccntractor ,  what  constitutes  a  settl  e;:.:ne  . 
what  ethics  are  involved  in  negotiations,  how  to 
elicit  compliance  with  the  Truth  in  Negotiations  Acc  and  the 
disclosure  of  unallowable  costs,  and  "does  'law'  protect  the 
government  from  poor  ethics  on  contractor's  part?". 

Topic  4 .  The  topic  of  pricing  support  for  source 
selection  should  include  the  following  subtopics:  methods  of 
evaluation;  defining  realistic,  complete,  and  reasonable;  how 
to  perform  cost  realism  assessments  with  limited,  uncertified 


cost  and  pricing  data;  how  to  handle  uncorrected  Deficiency 
Reports;  and  the  rules  of  source  selection. 

Topic  5 .  To  establishing  a  training  program  £wr 
subordinates,  technical  training  in  such  areas  as  price/cost 
analysis,  cost  estimating,  contract  administration,  cost 
accounting  standards,  overhead  analysis,  and  so  forth  must  be 
addressed . 

Topics  6  (also  #19 ;  .  The  topic  of  roles  and 
responsibilities  of  pricers  in  negotiations  is  closely  relat¬ 
ed  to  the  topic  which  ranked  nineteenth  overall,  program 
management/pricing  conflict  resolution,  and  which  is  polar¬ 
ised.  What  is  being  sought  is  a  clearer  definition  of  the 
mission  and  role  of  the  pricer,  the  contracting  officer, 
the  auditor,  other  technical  support  personnel,  and  of  the 
program  manager.  Those  roles  are  defined  by  each  particular 
office's  organizational  structure  and  cultural  norms.  7cr 
example,  one  purchasing  center  repcraedly  exhihias  ccnflict 
between  the  program  managers  and  the  pricers,  v,horeas  such 
conflicts  seldom  if  ever  occur  at  other  purchasing  centers. 
The  normis  of  the  latter  offices  have  more  clearly  u-mlirs'd 
roles  for  each  function.  In  addition  to  clearer  definitions, 
techniques  are  needed  to  resolve  conflicts  whenever  they 
occur . 

Topic  7 .  Refer  to  topic  #3. 

Topic  8 .  There  were  polarized  views  regarding 


whether  the  topic  of  factfinding  should  be  considered  for  an 


executive  pricing  course.  On,  the  "no”  side  are  those  who 
feel  that  either  the  pricer  is  just  one  of  a  factfinding  rea.T. 
(not  the  leader)  or  that  factfinding  is  a  basic  task  already 
mastered  by  all  who  are  required  to  take  the  course.  Cn  the 
"yes"  side  are  those  who  feel  that  the  pricer  is  the  key 
player  in  factfinding  and  that  inadequate  attention  has  been 
given  to  ensuring  that  all  elements  of  the  statement  of  work 
are  addressed  and  appropriately  priced  in  the  contractor's 
proposals.  Accordingly,  the  recommended  subtopics  are  whac 
to  look  for  and  how  to  conduct  the  meetings. 

Topic  9 .  Obtaining  adequate  price  competition  for 
other  than  firm  fixed  price  contracts  requires  structuring 
the  evaluation  criteria  to  obtain  an  "apples-to-apples" 
comparison.  Requested  are  examples  for  fixed  price  incentive 
firm,  cost  plus  incentive  fee,  cost  plus  fixed  fee,  and  time 
and  material  contract  types. 

Tcpi  'j  10 .  Some  pricing  offices  have  beccme  in¬ 
volved  in  the  evaluation  of  co.v.p eti ti ve  proposals  submitted 
various  depots  and,  sometimes,  private  entities.  To  evaluate 
the  proposals,  a  better  understanding  of  the  rel ati cnship 
between  the  Cost  Comparability  Handbook  and  the  Cost  Account¬ 
ing  Standards,  instruction  on  how  to  perform  meaningful 
comparisons  of  public  and  private  proposals,  and  an  overview 
of  differences  in  costing  practices  among  the  services  are 


needed . 
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Summary 


Presentad  in  this  chapter  were  the  quantitative  results 
of  the  student  surveys  and  quantitative  and  qualitative 
results  of  the  expert  questionnaires.  A  detailed  analysis  of 
each  set  of  data  was  also  presented.  The  result  of  this 
analysis  is  a  prioritized  list  of  recommended  topics  to  be 
taught  in  COH  331  --  "Executive  Cost  and  Price  Analysis," 
reflecting  a  consensus  by  the  panel  of  experts,  as  well  as 
specific  comments  on  the  top  ten  recommended  topics. 
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chapter  V 

Conclusions  and  Recommendations 


Overview 

This  final  chapter  summarizes  Chapters  I  through  III  and 
the  findings  set  forth  in  Chapter  IV.  The  practical  implica¬ 
tions  of  the  research  are  discussed.  Recommendations  for 
studies  which  replicate  or  follow-up  this  effort  are  made. 

Summary  of  Preceding  Chanters 

General  Issue .  In  order  to  address  the  education  relat¬ 
ed  findings  of  the  President’s  Blue  Ribbon  Commission  on 
Defense  Management  (Packard  Commission),  the  Department  of 
Defense  established  a  three  tiered  certification  process  for 
acquisition  personnel.  Each  certification  level  contains 
minimum  requirements  for  education,  traini.ag,  and  enperience. 
Several  Air  Force  Institute  of  Technology  Professional  Con¬ 
tinuing  Education  departments  are  conducting  studies  to 
determine  how  best  to  fulfill  the  various  training  require¬ 
ments  for  the  different  certification  levels. 

Specific  Problem.  Air  Force  Institute  of  Technology 
Professional  Continuing  Education  course  CON  331  --  "Execu¬ 
tive  Cost  and  Price  Analysis"  is  now  required  for  Level  III 
certification,  for  pricing  managers  and  executives.  What 
changes  in  the  existing  course  curriculum  need  to  be  made  to 
meet  the  needs  of  the  current  students? 
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Investigative  Questions .  In  order  to  resolve  the  spe¬ 
cific  problem,  the  following  investigative  questions  have 
been  answered  throui^h  literature  reviews  and  use  of  the 
Delphi  technique  with  a  carefully  chosen  panel  of  experts 
representing  the  population  required  to  take  the  course. 

1 .  Who  is  present  1 y  required  to  take  the  course? 
Any  individual  in  the  contracting  career  field  (GS-11C2  and 
comparable  military)  who  has  the  grade  of  GS/GM-13  or  above 
or  the  rank  of  0-4  or  above  and  holds  a  job  that  is  primarily 
cost  and  price  analysis  oriented  is  required  to  have  complet¬ 
ed  CON  331  --  "Executive  Cost  and  Price  Analysis" 

(DOD  1991;  3.1-5) . 

What  are  the  current  course  objectives  and 
content?  The  current  course  description  states  that  it 
"Provides  experienced  procurement  analysts  with  statistics 
and  accounting  tools  necessary  to  analyse  costs,  establisn 
cost  estimating  relationships,  and  develop  reasonable  c-si 
estimates  in  large  system  procurements"  (AFIT  1939:  . 

Topics  in  the  current  course  include  the  impact  of  unallowa¬ 
ble  pool  and  base  costs  on  overhead  allocation,  probability 
and  decision  analysis,  tracking  actual  costs  to  estimated 
costs  to  ascertain  bias,  and  risk  analysis  and  production 
quantity. 


3 .  What  specific  topics  represent  the  needs  of  the 
peopi  e  who  are  r  c-gui  rad  to  take  t:.a  course?  In  rank  order, 
from  most  desirable  to  least  desirable,  the  recoiT^jnended 
topics  for  the  Executive  Pricing  Course  are  listed  in  Table 

5.1. 


TABLE  5.1 

RECOMMENDED  TOPICS 


1.  Incentive  Arrangements 

2.  Current  and  Pending  Laws,  Court  Gases, 

Regulations 

3.  Ethical  Conduct 

4.  Source  Selection  Pricing  Support 

5.  Establishing  a  Training  Program  for 

Subordinates 

D,  Roles  and  Responsibilities  of  Pricers  in 
Negotiations 

7.  Ethical  Settlements  in  Negotiations 
3.  Factfinding 

9.  Adequate  Price  Competition  for  Other  than  FF? 
Contracts 

10.  Depot  Competition 

11.  Requests  for  Equitable  Adjustments 

12.  Estimating 

13 .  Claims 

14.  Overhead  (Unallowables) 

15.  Specialised  Reviews,  e.g..  Should  Cost 

16.  Fundamentals  of  Contract  Theory 

17.  Cost  Accounting  Standards 

18.  Pricing  of  Commercial,  Nondev  el  cpmental  1 1 1;... 
15.  Program  Management/Pricing  Conflict  Resclutic.. 

20.  Estimates  vs  Actuals 

21.  Department  of  Defense  Profit  Policy 

22.  Contractor  Financing 

23.  Eichleay  Formula  for  Unabsorbed  Overhead 

24.  Impact  and  Cost  of  C/SCSC,  CPR,  C/'SSR,  etc. 

25.  Cost  Monitoring  Reviews 
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Conclusions 

The  change  in  clientele  for  the  "Executive  Cost  and 
Pricing  Course"  has  resulted  in  a  need  to  change  the  topics 
addressed  in  the  course.  While  those  journeymen  analyst  who 
occasionally  take  the  course  express  appreciation  for  and 
satisfaction  with  the  current  topic  selection,  the  pricing 
managers  and  executives  who  complete  the  current  course 
generally  express  dissatisfaction  with  the  content.  The 
panel  of  experts  recommended  the  topics  listed  in  Table  5.1 
be  considered  for  the  course.  In  considering  their  recommen¬ 
dations,  the  course  instructor  may  choose  to  group  the  topics 
in  a  manner  that  facilitates  their  presentation. 

Recommendations 

Comments  on  the  methodology.  Use  of  the  Delphi  tech¬ 
nique  with  Kendall’s  Coefficient  of  Concordance  has  been 
quite  effective  and  practical.  Several  changes,  thcug:.,  are 
recommended  to  enhance  its  usefulness.  In  the  first  round 
the  experts  should  1)  rank  the  topics,  2)  define  the  most 
preferred  five  and  explain  why  each  is  important,  and  3)  list 
and  define  any  additional  topics  and  explain  why  each  is 
important,  but  do  not  rank  the  additional  topics. 

Before  the  second  round,  any  topic  listed  by  all  experts  * 

as  being  in  the  bottom  50%  could  be  dropped.  This  would  make 
successive  rounds  less  cumbersome  and  would  not  affect  the 
topics  selected  for  the  course  since  the  available  time 
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typically  limits  the  number  of  topics  to  be  covered  to  a 
maximum  of  about  two  to  four  per  week. 

In  the  second  round  the  experts  should  again  rank  all 
the  topics,  including  the  topics  added  by  the  experts  in  the 
first  round.  The  definitions  and  explanations,  as  well  as 
the  composite  ranking  of  the  first  round,  should  be  provided 
as  information  in  the  second  round. 

Saaoestions  for  Foil ow-On  Studies .  The  Air  Force  Lepucy 
Undersecretary  for  Acquisition  has  expressed  an  interes..  xii 
studies  which  address  the  extent  to  which  required  Profes¬ 
sional  Continuing  Education  courses  meet  the  Acquisition 
Professional  Development  Program  educational  goals.  Accord¬ 
ingly,  similar  studies  for  all  required  Professional  Coiicsnu 
ing  Education  courses  are  warranted.  With  a  methodology 
already  established,  future  research  could  address  gr^Ot.^  or 
courses  rather  than  one  course  at  a  time.  After  a 
CON  331  "Executive  Cost  and  Price  Analysis"  has  been  oil. 
to  several  classes,  a  follcw-up  study  on  CON  331  should  bo 
conducted  to  ascertain  whether  the  recommended  topics  ude 
quateiy  address  the  training  needs  of  pricing  managers  and 
executives . 

Summary 

The  Department  of  Defense  established  CON  331  -- 
Executive  Cost  and  Price  Analysis  as  a  prerequisite  for  Level 
III  certification  and  Level  III  certification  is  required  of 
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all  pricing  managers  and  executives.  The  current  course 
content  was  designed  to  meet  the  needs  of  pricing  journeymen 
or  "master”  pricers,  covering  advanced  techniques  of  cost 
analysis  and  risk  assessment.  Therefore,  the  current  course 
content  does  not  adequately  address  the  needs  of  the  new 
clientele.  The  result  of  this  study  has  been  a  recommended 
selection  of  topics  for  the  Professional  Continuing  Education 
course  CON  331  --  Executive  Cost  and  Price  .Analysis. 

The  Department  of  Defense  is  interested  in  determining 
the  extent  to  which  the  Professional  Continuing  Education 
courses  address  the  training  needs  of  acquisition  personnel . 
To  that  end  additional  studies  are  recommended  which  examine 
the  effectiveness  of  various  Professional  Continuing  Educa¬ 
tion  courses  in  meeting  the  education  goals  of  the  Acquisi¬ 
tion  Professional  Development  Program. 


Appendix  A 
The  Delphi  Method 


When  to  Use 

The  Delphi  technique  is  a  decisionmaking  aid  that  elic¬ 
its  a  consensus  judgement  or  opinion  from  a  panel  of  experts 
through  an  iterative  questionnaire  and  feedback  process.  The 
method  is  most  useful  whenever  insufficient  or  no  applicable 
data  exits,  the  required  data  is  too  expensive  to  obtain  and 
analyze,  or  the  significant  problem  variables  and  their 
interaction  are  not  clearly  known  (Zaloom  1974:  32). 

The  Process 

The  Delphi  process  involves  sending  a  questionnaire  to  a 
panel  of  experts  who  remain  anonymous  during  the  survey 
process.  The  results  of  the  questionnaire  are  tabulated  and 
resubmitted  to  the  panel  for  reconsideration.  Experts  are 
asked  to  explain  their  opinions  so  that  the  entire  panel  can 
use  all  available  information  and  judgements  during  reconsid¬ 
eration.  The  revised  submissions  are  tabulated  and  submitted 
again  to  the  experts.  This  process  of  submission,  tabula¬ 
tion,  explanation,  and  resubmission  is  repeated  until  a 
consensus  is  reached  or  until  divergent  views  are  clarified 
and  well  understood  by  the  panel  members,  the  researchers, 
and  the  decis.onmakers . 
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Uses  of  Delphi  and  Current  Application 


The  primary  use  of  the  Delphi  method  has  been  in  fore¬ 
casting,  e.g.,  market  prospects  or  technology  development. 
More  recently  the  technique  has  been  used  in  policy  formula¬ 
tion  in  areas  of  education,  transportation,  health  care 
(Rohrbaugh  1979:  76),  urban  planning  (Morgan,  Pelissero,  and 
England  1979:  380),  and  productivity  assessments  (Scott  and 
Simmons  1974:  71),  and  in  predicting  the  impact  of  socioeco¬ 
nomic  programs,  of  changes  in  department  stores,  of  land  use 
policy,  and  of  various  information  systems  (Tersine  and  Riggs 
1976:  52).  Miles  Weaver  (1988:  18),  Martin  (1993:  267),  and 
Dorothy  Olshfski  and  Alma  Joseph  (1991:  297)  advocate  the  use 
of  the  Delphi  technique  in  training  need  assessment  and 
curriculum  design.  Usually,  a  consensus  is  reached  in  four 
or  fewer  rounds  (Zaloom  1974:  33).  However,  the  education 
literature  has  documented  consensus  in  three  rounds  (Weaver 
1988:18) . 

Advantages 

Delphi  is  recognized  to  have  several  advantages  over 
other  group  decisionmaking  techniques  or  other  more  conven¬ 
tional  means  of  gathering  opinions  on  topics  not  subject  to 
precise  quantification.  Since  the  experts  are  anonymous,  one 
may  change  his  mind  without  ego  involvement;  each  participant 
is  less  subject  to  being  unduly  influenced  by  the  status  or 
power  of  another  expert  participant;  there  is  reduced  pressure 
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to  experts  to  be  influenced  by  the  bandwagon  effect  (Tersine 
and  Riggs  1976:  51-52).  There  is  no  chance  that  a  partici¬ 
pant  with  strong  oratory  skills  can  sway  the  panel  and  the 
panelists  more  readily  abandon  popular  opinions  that  are  not 
well  founded  (Zaloom  1974:  32).  Other  advantages  include  low 
cost,  unnecessary  to  arrange  common  meeting  times  and  places, 
no  restrictions  on  the  number  of  participants,  it  encourages 
participant  consideration  of  alternative  opinions  (Tersine 
and  Riggs  1976:  52),  and,  as  with  other  group  techniques,  the 
panelists  have  a  sense  of  ownership  toward  the  final  product, 
which  is  important  in  educational  uses  (Weaver  1988:  18). 

The  survey  is  also  easily  and  accurately  interpreted  by  the 
experts.  With  a  relatively  small  number  of  experts  compris¬ 
ing  the  panel,  it  is  relatively  convenient  to  execute  and  is, 
hence,  easily  replicated. 

Disadvantages 

The  Delphi  technique  is  not  without  some  limitations. 

The  panel  selection  is  a  critical  step;  poor  panel  selection 
can  make  the  whole  process  worthless;  the  number  of  calendar 
days  required  to  complete  each  iteration  of  questionnaire  can 
be  three  or  four  weeks,  especially  if  the  experts  are  pressed 
for  time  by  other  commitments  and  must  "work  in"  the  ques¬ 
tionnaire  responses;  and  expert  motivation  can  wane  (Tersine 
and  Riggs  1976:  55).  Sackman,  cited  by  Morgan  and  others, 
reviewed  150  Delphi  studies  and  found,  among  other  things. 


A-3 


that  it  does  not  attain  "professional  standards  for  question¬ 
naire  design  and  validation,"  inhibits  individuality  and 
adversarial  processes,  misleads  the  uninformed  regarding  its 
precision,  and  fosters  a  form  of  elitism  in  solving  social 
problems  (Morgan,  Pelissero,  and  England  1979:  381). 

Some  of  Sackman's  concerns  beg  the  question  of  the 
Delphi  technique;  if  more  rigorous  research  techniques 
wherein  statistical  tests  and  other  professional  standards 
could  be  employed  were  possible,  then  the  Delphi  technique 
would  be  set  aside  in  favor  of  the  more  rigorous  methods. 
Further,  most  of  Sackman's  concerns  were  directed  toward 
forecasts  in  technology,  not  the  area  of  curriculum  develop¬ 
ment.  As  stated,  Delphi  has  been  identified  as  a  preferred 
method  for  the  use  employed  in  this  research. 

Requirements  of  Experts  and  Panel  Size 

The  size  and  composition  of  the  panel  of  experts  varies 
among  areas  of  use.  Panels  have  ranged  in  size  from  four 
(Baylock  and  Rees  1984;  78)  to  421  (Tersine  and  Riggs  1976: 
54).  With  more  homogeneous  groups,  smaller  panels  are  needed 
(Tersine  and  Riggs  1976:  54).  The  experts  must  have  knowl¬ 
edge  of  the  problem  area,  have  a  good  performance  record  in 
their  areas  of  expertise,  have  a  high  degree  of  objectivity 
and  rationality,  be  available  and  commit  the  time  required 
for  the  study  (Tersine  and  Riggs  1976:  53-54).  For  education 
needs  assessment,  the  experts  should  also  be  broadly  represen- 
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tative  of  the  population  that  is  being  assessed  (Olshfski  and 
Joseph  1991:  298).  Olshfski  and  Joseph  further  distinguish 
using  Delphi  for  educational  needs  assessment  versus  fore¬ 
casting  in  the  operational  definition  of  expert.  They  note 
that  in  forecasting  the  expert  is  defined  in  terms  of  creden¬ 
tials  and  experience.  For  the  educational  needs  assessment 
Delphi,  they  use  "Mary  Parker  Follett's  (1924)  definition  of 
expert:  the  individual  who  has  experience  in  the  organiza¬ 

tion  and  who  therefore  knows  more  about  life  inside  the 
organization  than  anyone  on  the  outside"  (Olshfski  and  Joseph 
1991:  298). 
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CON  331  (WQKT  540)  --  QUESTIONNAIRE 

Advanced  Ccxitract  Pricing,  WQHT  540,  is  an  executive  level  pricing 
course  required  for  APDP  level  3  certification.  WQbfT  540  was 
originally  designed  to  provide  "joumeynan”  and  "master"  price 
analysts  quantitative  techniques  necessary  for  working  the  more 
conplex  cases  oicountered.  As  such  the  current  course  ccntent  does 
not  address  areas  that  may  be  of  more  value  to  the  pricing  manager  or 
executive.  Accordingly,  this  questionnaire  seeks  to  ascertain  those 
topics  you  consider  more  helpful  to  the  pricing  rretrsu^r  or  executive. 

Please  provide  the  information  requested  below.  Mail  to: 

DEPARTMEKT  OF  THE  AIR  FORCE 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY/ LSQ 
Attn:  JEFF  DANEMAN 
2950  P  STREET 

WRIGHT-PATTERSCW  AFB  CH  45433-7765 
Thank  you  for  your  time  and  assistance. 

Background  information: 

1.1  Current  position:  _ 

1.2  Months  in  current  position:  _ 

1.3  Grade  or  Rank:  _ 

1.4  Years  of  experience  at  indicated  supervisory/ executive  level: 

Not  a  supervisor:  _ 

First  level:  years 

Second  level:  years 

Third  level:  years 

Staff:  years  Where:  _ 


Listed  on  the  following  pages  ar^  veral  topics.  For  each  please 
indicate  the  need  for  that  topic  to  oe  covered,  from  a  low  of  i  to  a 
high  of  5.  With  any  topic  your  coninents  are  solicited. 


TOPIC 


NEED 

Low 


2.1  Deferred  Coipensation 

2.2  Unconrpensated  Overtime 

2.3  Estimates  vs  Actuals 

2.4  Incentive  Arrangements 

2 . 5  Depreciation 

2.6  Evaluation  of  Bill  of  Materials 

2.7  Lease  vs  Buy  Decisions 

2 . 8  Overhead  (Una 1 1 owabl es ) 

2 . 9  Estimating 

2.10  Warranties 

2.11  Cost  Accounting  Standards 

2.12  Ethical  Settlements  in  Negotiations 

2.13  Current  and  Pending  Laws,  Court  Cases,  and 

Regulations 

2.14  Ethical  Conduct 

2.15  Defective  Pricing 

2.16  Defense  Base  Operating  Fund 

2.17  Statistical  Tools  for  Pricing 

2.18  Acquisition  Professional  Development  Program 

2.19  Fundamentals  of  extract  Theory  (for  those  with 

little  buying  experience) 

2.20  Impact  and  Cost  of  C/SCSC  (CPR),  C/SSR, 

Mil  Std  1567A,  etc. 


12  3 

12  3 
12  3 

12  3 

12  3 
12  3 

12  3 

12  3 

12  3 

12  3 

12  3 

12  3 

12  3 

12  3 
12  3 

12  3 

12  3 

12  3 

12  3 

12  3 


2.21  Establishing  a  Training  Program  for  Subordinates  123 

2.22  Available  Management  Information  Systems  123 

2.23  Requests  for  Equitable  Adjvetment  12  3 

2.24  Claims  12  3 


High 
4  5 
4  5 
4  5 
4  5 
4  5 
4  5 
4  5 
4  5 
4  5 
4  5 
4  5 
4  5 
4  5 

4  5 
4  5 
4  5 
4  5 
4  5 
4  5 

4  5 

4  5 
4  5 
4  5 
4  5 
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TOPIC 


Low 


^^SED 


2.25  Disputed  Claiins 

2.26  Alternative  Disputes  Resolution 

2.27  Court  i^pearances 

2.28  Value  Engineering  Change  Proposals 

2.29  Voluntary  Refunds 

2.30  Termination  for  Convenience 

2.31  Termination  for  Default 

2.32  Depot  Competition 

2.33  Factfinding 

2.34  Analysis  of  Technical  Reports/ Input 

2.35  Analysis  of  Audit  Reports/ Input 

2.36  Source  Selection  Pricing  Support 

2.37  Foreign  Military  Sales  Pricing  Support 


High 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


IN  THE  FOLLOWING  SPACES,  LIST  ANY  ADDITIONAL  TOPICS  YOU  BELIEVE 
SHOULD  BE  OOTJSIDERSD.  FOR  EACH,  PLEASE  INDICATE  THE  LEVEL  OF  NEEL. 


3.1 


1  2  3  4  5 


3.2 


1  2  3  4  5 


3.3  1  2  3  4  5 

3.4  1  2  3  4  5 


3.5 


1  2  3  4  5 


3.6 


1  2  3  4  5 
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Appendix  D 

Initial  Expert  Questionnaire 

DEPARTMENT  OF  THE  AIR  FORCE 

AM  UNWBISfTV 

AM  KMCC  MtTITUn  'mCHNOLOGV 


30  December  1992 


non:  AFIT/LSQ  (J.  Daneman  DSN  785-8410) 

2950  P  Street 
WPAPB  OH  45433-7765 

SUBJECT:  Executive  Pricing  Course  Questionnaire  #1 

TO:  [addrescode]  ([firstname]  [lastname]  [ commphone ] ) 


Thank  you  for  agreeing  to  help  reorient  the  Executive  Pricing 
Course  which  is  required  for  APDP  Level  III  certification  for 
price  analysts.  The  course  was  originally  designed  for 
journeyman  analysts  and  needs  to  be  rewritten  to  address  issues 
that  are  more  pertinent  to  pricing  managers  and  executives. 

Two  i.FIT  graduate  students,  Mike  Grove  and  Eddie  Upshaw,  ^re 
developing  a  method  to  update  the  course  which  relies  on  the 
expertise  of  you  and  five  other  experts  in  the  field.  With  your 
help,  we  hope  to  reach  a  consensus  regarding  the  topics  and 
subtopics  that  should  be  covered  in  the  course.  Each  expert 
will  be  sent  several  brief  questionnaires,  each  narrowing  and 
prioritizing  topics  and  subtopics  until  a  consensus  is  reached. 

While  any  change  to  this  course  is  subject  to  approval  by  the 
Defense  Acquisition  University,  it  is  hoped  this  research  will 
serve  as  the  basis  for  a  rewrite  of  the  course  and  as  the  basis 
for  a  similar  course  to  be  taught  in  AFIT's  graduate  program. 
The  graduate  students  doing  the  "f..otwork"  for  this  research 
will  base  their  thesis  upon  the  research  effort. 

The  initial  effort  consists  of  your  responses  on  the  attached 
questionnaire.  Once  we  have  received  and  reviewed  all  the 
responses,  another  updated  questionnaire  will  be  forwarded  to 
you.  If  you  have  any  questions,  you  can  contact  me  at  DSN 
785-8410. 


Jeff  Daneman,  Course  Director 
Air  Force  Institute  of 
Technology 


2  Atch 

1.  Questionnaire 

2.  Return  Envelope 


CON  331  (WQMT  540)  —  QUEST i ONNA ! RE 


Advanced  Contract  Pricing,  WC!(^  54C.  is  an  executive  level  ortcing 
course  reouired  for  APDP  level  3  certification.  WQMT  540  was 
originally  designed  to  orovice  "  jourr.evnan"  arc  "master’’  or’ce 
analysts  quant i native  techniques  necessary  for  working  the  more 
carp  lex  ceises  encountered.  As  such  the  current  course  content  does 
not  address  areas  that  .nnay  be  of  more  value  ~o  cne  or  icing  manager  cr 
executive.  Accordingly,  tn-ss  questionnaire  seeks  to  ascertain  those 
tooics  you  consider  more  heloful  to  the  pricing  manager  or  executive. 

Please  provide  the  information  requested  below.  Mail  to: 

DEPARTMENT  OF  AIR  '^CRCE 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY /LSQ 
Attn:  JEFF  DANEMAN 
2950  P  STREET 

WR10HT-PATTERSON  AFB  OH  45433-7755 
Thank  vou  for  vcur  time  and  assistance. 

Sackground  i nf crmat i on : 

'.I  Current  ocsition:  _ 

1.2  Months  in  current  posit 'on:  _ 

- . 3  Grade  or  Rank :  _ 

1.4  Years  of  exoer'ence  at  "indicated  suoerviscrv/execut've  ’eve:: 

Not  a  suoervisor: 

years 
years 
years 

veans  W'nere:  _ 


uisted  on  the  following  pages  are  several  topics.  Fcr  each  o’ease 
indicate  the  need  for  that  topic  to  be  covered,  from  a  low  cf  ■;  to  a 
high  of  5.  With  any  topic  your  comT:e.nts  are  solicited. 


‘^'irst  level: 
Second  level: 
Third  level: 
Staff: 


D-2 


TOPIC  N£ED 

LOW 


2.1 

Deferred  Ccrroensation 

1 

2 

3 

4 

5 

2.2 

Uncoroensarec  Overt  irre 

1 

i 

2 

3 

A 

5 

2.3 

Estimates  vs  Actuals 

O 

im. 

3 

4 

5 

2.4 

'  1  r.cer.t  i  ve  Arrangements 

*5 

2 

3 

4 

5 

2.5 

Depreciation 

1 

2 

3 

4 

2.6 

Evaluation  of  3il''  of  Materials 

O 

3 

4 

5 

2.7 

Lease  vs  3uv  Dec  "is  ions 

1 

2 

3 

4 

c 

2.8 

Overhead  ( Ura 1 1 owab 1 es ) 

1 

2 

-3 

4 

5 

2.9 

Estimating 

1 

2 

3 

4 

5 

2. 1C 

Warranties 

■1 

2 

3 

A 

c 

2.11 

Cost  Accounting  Standards 

1 

o 

3 

c 

2.12 

Ethical  Settlements  in  Megotiations 

1 

o 

3 

.1 

5 

2.13 

Current  and  oending  Laws.  Court  Cases,  and 

Regu ' at ■ ons 

1 

o 

3 

A 

5 

2.14 

Ethical  Conduct 

1 

2 

3 

.1 

5 

2.15 

Defect "ive  ^n'c'ng 

o 

3 

4 

5 

2.16 

Defense  3ase  Ooerating  Fu.nd 

n 

1 

= 

2.^7 

Statistical  Tools  for  P'-icinc 

o 

3 

5 

2.18 

Accuisiticn  Professional  Develoomenc  Program 

•1 

n 

■5 

- 

5 

2.19 

Fundamentals  of  Contract  Theory  (for  t.hose  wich 
little  buy i ng  exoer i ence ) 

1 

n 

3 

4 

c 

w 

2.20 

impact  and  Cost  of  C/SCSC  (CPR),  C/SSR. 

Mil  Std  1567A,  etc. 

< 

2 

5 

n 

5 

2.21 

Establishing  a  Training  Progran  for  Subordinates 

1 

2 

3 

4 

c 

2.22 

• 

Available  Management  information  Systems 

1 

2 

3 

4 

5 

2.23 

Reauests  for  Equitable  Adjustment 

1 

2 

3 

4 

5 

2.24 

Claims 

1 

2 

3 

4 

5 
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TOPIC 

2.25  Disouted  Claims 

2.26  Alternative  Disputes  Resolution 

2.27  Court  Appearances 

2.28  Value  Engineering  Change  Prooosals 

2.29  Voluntary  Refuncs 

2.30  Termination  for  Convenience 

2.31  Termination  for  Default 

2.32  Depot  Corrcetition 

2.33  Factfinding 

2.34  Analysis  of  "Technical  Reoorts/lnout 

2.35  Analysis  of  Audit  Reports/ 1  .ncut 

2.36  Source  Selection  Pricing  Support 

2.37  Foreign  Military  Sales  Pricing  Support 


NEED 

Low  High 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

12  3  4  5 

1  O  0  C 

•  (ho  <->r 

1  2  3  4  5 

1  2  3  4  5 


IN  THE  FOLLCA/iNG  SPACES.  _ :  S”  ANY  ADD!“:ONAL  TOPICS  YOU  BELIEVE 
SHOULD  BE  CONSIDERED.  FOR  EAC-.  PLEASE  INDICATE  THE  LEVEL  0"  NEED. 

3.1  1  2  3  4  5 


3.; 


3.3 


,3.4 

3.5 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


1  2  3  4  5 


3.6 


1  2  3  4  5 
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Initial  Expert  Questionnaire  Results 


Topics 

Raw 

Data 

Scores 

1 

Frequency 

2  3  4  5 

3.12 

Ethical  Negot 

5 

5 

4 

3 

5 

5 

0 

0 

1 

1 

4 

3.13 

Laws,  Cases 

4 

5 

3 

5 

5 

4 

0 

0 

1 

2 

3 

3.14 

Ethical  Conduct 

5 

5 

4 

3 

5 

4 

0 

0 

1 

2 

3 

3.8 

0/H  (Unallou) 

4 

4 

3 

1 

5 

5 

1 

0 

1 

2 

2 

3.9 

Estimating 

4 

2 

5 

1 

5 

4 

1 

1 

0 

2 

2 

Depot  Competition 

4 

5 

4 

5 

1 

1 

2 

0 

0 

2 

2 

3.19 

Contract  Basics 

2 

5 

3 

5 

4 

4 

0 

1 

1 

2 

2 

3.4 

Incentive  Arrang 

4 

4 

4 

4 

4 

5 

0 

0 

0 

5 

1 

3.20 

CPR,  1557A 

4 

5 

3 

4 

4 

2 

0 

1 

1 

3 

1 

3.21 

Subord  Train 

4 

4 

4 

4 

4 

4 

0 

0 

0 

6 

0 

Factfinding 

3 

2 

5 

1 

5 

5 

1 

1 

1 

0 

3 

3.11 

CAS 

4 

4 

2 

1 

5 

3 

1 

1 

1 

2 

1 

3.3 

Est  vs  Act 

3 

3 

4 

1 

5 

4 

1 

0 

2 

2 

1 

3.23 

REA 

4 

4 

3 

1 

2 

5 

1 

1 

1 

2 

1 

Source 

Selection 

3 

4 

4 

4 

3 

3 

0 

0 

3 

3 

0 

3.24 

Claims 

4 

4 

3 

1 

2 

4 

1 

1 

1 

3 

0 

3.26 

ADR 

3 

5 

2 

1 

3 

3 

1 

1 

3 

0 

1 

3.15 

Defective  Price 

3 

5 

3 

1 

3 

4  • 

1 

0 

3 

1 

1 

PMS  Pricmg  Support 

3 

4 

3 

1 

3 

5 

1 

0 

3 

1 

1 

3.6 

Eval  of  BOM 

2 

3 

3 

1 

3 

3 

1 

1 

4 

0 

0 

3.17 

Stat  Tools 

3 

3 

4 

2 

4 

2 

0 

2 

2 

2 

0 

3.27 

Court  Appear 

3 

4 

2 

1 

4 

1 

2 

1 

1 

2 

0 

3.5 

Depreciation 

3 

3 

2 

1 

3 

4 

1 

1 

3 

1 

0 

3.10 

Warranties 

3 

3 

3 

1 

1 

3 

2 

0 

4 

0 

0 

3.25 

Disputed  Claims 

3 

4 

2 

1 

3 

2 

1 

2 

2 

1 

0 

3.22 

MIS 

3 

4 

3 

3 

3 

2 

0 

1 

4 

1 

0 

3.1 

Deferred  Comp 

4 

3 

2 

1 

4 

1 

2 

1 

1 

2 

0 

3.2 

Uncomp  0/T 

4 

5 

2 

1 

1 

1 

3 

1 

0 

1 

1 

3.31 

T  for  D 

3 

2 

2 

1 

3 

1 

2 

2 

2 

0 

0 

3.29 

Volunt  Refunds 

4 

4 

2 

1 

2 

1 

2 

2 

0 

2 

0 

3.16 

DBOF 

4 

2 

3 

1 

1 

1 

3 

1 

1 

1 

0 

3.28 

VECP, 

3 

3 

2 

1 

2 

1 

2 

2 

2 

0 

0 

3.30 

T  for  C 

3 

2 

2 

1 

3 

1 

2 

2 

2 

0 

0 

3.7 

Lease  vs  Buy 

3 

2 

2 

1 

2 

4 

1 

3 

1 

1 

0 
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Second  Expert  Questionnaire 

DEPARTMENT  OF  THE  AIR  FORCE 

AMUMWERaiTV 

AMKRCSmSTtlUTE  OFTCCHMOUXIV 


F»0H:  AFIT/LSQ  (J.  Daneman  DSN  785-8410) 

2950  P  Street 
WPAFB  OH  45433-7765 

SUBJECT:  Executive  Pricing  Course  Questionnaire  #1 

TO;  [addrescode]  ([firstname]  [lastname]  [commphone]) 


Thank  you  for  responding  to  the  first  questionnaire  designed  to 
update  the  "Executive  Cost  and  Price  Analysis"  course.  The 
responses  were  tallied  and  those  topics  receiving  a  rating  of 
three  or  less  from  50515  of  the  expert  panel  have  been  dropped 
from  further  consideration.  The  remaining  sixteen  topics  are 
listed  on  the  enclosed  questionnaire  in  the  order  of 
preference,  from  highest  to  lowest,  based  upon  a  mean  of  the 
ratings  assigned  by  the  panel.  Also  listed  are  nine  additional 
topics  which  were  suggested  by  either  yourself  or  other  panel 
members . 

Please  rank  all  the  topics  listed  in  the  questionnaire  from  one 
to  twenty-five,  with  one  being  the  most  needed,  two  being 
second-most  needed  and  so  on  down,  with  twenty-fifth  being  the 
least  needed  topic.  Afterwards,  please  list  the  subtopics  and 
points  of  emphasis  you  feel  should  be  included  in  the  classroom 
discussions  for  at  least  the  five  topics  you  consider  most 
important . 

Delays  in  the  pouch  mail  system  added  about  four  weeks  to  the 
time  required  for  delivery  of  the  first  questionnaire.  To 
speed  things  up,  franked  return  envelopes  are  enclosed  for  your 
use.  Please  ensure  they  are  delivered  through  the  U.  S.  Postal 
System.  If  possible,  please  respond  by  31  Mar  93. 

If  a  third  iteration  is  necessary,  you  should  receive  a 
questionnaire,  which  will  be  similar  to  this  one,  during  the 
first  week  of  April.  If  a  third  iteration  is  not  necessary, 
you  will  be  sent  a  summary  of  the  anonymous  responses  provided 
by  all  the  panel  members.  If  you  have  any  questions,  you  can 
contact  me  at  DSN  785-8410. 


Jeff  Daneman,  Course  Director  2  Atch 

Air  Force  Institute  of  1.  Questionnaire 

Technology  2.  Return  Envelope 
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Rar.k  the  topics  listed  below  fron  1  to  25. 


RANK 


TOPIC 

Ethical  Sett  1  emer.ts  in  Negotiations 

Current  and  Pending  Laws.  Court  Cases,  and  Regulations 

Ethical  Conduct 

Incentive  Arrangements 

Establishing  a  Training  Prograr;  -or  Subordinates 

Fundamentals  of  Contract  Theory  (for  tnose  with  -itt'e 
buying  experience) 

Overhead  (Ural! owab 1 es ) 

Irrcact  ar.d  Cost  of  C/SCSC,  CPR,  C/SSR,  Mil  Std  1567a.  etc. 

Source  Selection  Pricing  Support 

Estimating 

Factfinding 

Estimates  vs  Actuals 

Deoot  Ccrroetition 

Cost  Accounting  Star.deirds 

Reouests  for  Ecuitable  Adjustments 

Claims 


Suggested  Topics  from  First  Questionnaire 

_  Adecuate  Price  Corroetition  for  Other  than  FFP  Arrangements 

_  Cont r actor  Financing 

_  Cost  Monitoring  Reviews 

_  DoD  Profit  Policy 

_  Eichleay  Formula  for  Unabsorbed  Overhead 

_  Pricing  of  Canmercial  Items.  Nondevelopmental  iters 

_  Program  Management /Pricing  Conflict  Resolution 

_  Roles  and  Responsibilities  of  Price  Analysts  in  .Negotiations 

_  Specialized  Cost  Reviews,  e.g..  Should  Cost  Reviews  or 

Subcontract  Pricing  Reviews 
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List  the  subtooics  and  points  of  emohasis  for  at  'east  the  five  tcs 
iTOortant  tonics. 

Topic  1 


* 


Tcoic  2 


Topic  3 


Tooic  4 


M 


Topic  5 
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Appendix  G 

Second  Expert  Questionnaire  Results 


Expert 

Tonic 

1 

0 

3 

4 

5 

6 

Ethical  Negotiations 

18 

3 

24 

16 

1 

3 

Current  Laws  and  Cases 

9 

2 

20 

13 

•> 

4 

Ethical  Conduct 

19 

4 

23 

6 

3 

5 

Incentive  Arrangements 
Establishing  A  Training  Program 

7 

10 

9 

1 

4 

9 

for  Subordinates 

13 

5 

10 

15 

5 

14 

Fundamentals  of  Contract  Theory 

8 

24 

15 

5 

6 

24 

Overhead  (Unallowables) 
Impact/Cost  of  CPR, 

14 

18 

5 

21 

7 

13 

Mil  Std  1567A,  etc 

21 

13 

18 

14 

20 

Source  Selection 

2 

15 

7 

4 

17 

15 

Estimating 

11 

19 

1 

12 

10 

21 

Factfinding 

12 

20 

2 

2 

11 

20 

Estimates  vs  Actuals 

10 

21 

4 

T  "> 

4m 

15 

Depot  Competition 

1 

14 

14 

7 

25 

11 

Cost  Accounting  Standards 
Requests  for  Equitable 

15 

23 

5 

11 

16 

12 

Adjustment 

4 

22 

16 

3 

^  a. 

6 

Claims 

Adequate  Price  Competition  for 

6 

17 

17 

8 

21 

7 

Other  than  FFP 

3 

11 

3 

17 

19 

10 

Contractor  Financing 

25 

12 

25 

20 

1  ■*' 

V 

Cost  Monitoring  Reviews 

24 

25 

13 

13 

13 

a.  C 

DoD  Profit  Policy 

Eichleay  Formula  for 

20 

3 

22 

19 

14 

1 

Unabsorbed  Overhead 

5 

16 

21 

23 

24 

17 

Pricing  of  Commercial  and  Ncn- 
devel opmental  Items 

Program  Management/Pricing 

16 

6 

11 

9 

23 

19 

Conflict  Resolution 
Roles/Responsibilities  of  the 

?•  T 

A.  ^ 

1 

19 

24 

18 

JL 

Price  Analyst  in  Negotiations 

17 

-  7 

3 

25 

9 

dL 

Specialized  Cost  Reviews 

23 

9 

12 

10 

8 

IS 
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Appendix  G  (Continued) 

Second  Expert  Questionnaire  Results 


Comments : 

Program  Management/Pricing  Conflict  Resolution 
-Program  management's  mission 
-Pricing '3  mission 
-Resolution  of  the  conflict 

Roles/Responsibilities  of  the  Analyst  in  Negotiations 
-Role  of  Contracting  Officer 
-Role  of  the  Price  Analyst 
-Role  of  the  field  agencies 
-Role  of  other  technical  support 

Ethical  Settlements  in  Negotiations 
-What  is  a  settlement? 

-What  ethics  are  involved  in  negotiations? 

-Do  laws  protect  the  Government  from  poor  ethics  on 
the  contractor's  part? 

-Equity  for  both  Government  and  contractor 

Current  Laws,  Court  Cases,  Regulations 
-FAR,  DFARS 
-ASBCA  cases 

Ethical  Conduct 

-Combine  with  "Ethical  Negotiations" 

-Truth  in  Negotiations  Act 
-Disclosure  of  unallowable  costs 

Incentive  Arrangements 
-Types  of  contracts 
-Performance  incentives 
-Tradeoffs 

Establishing  a  Training  Program  for  Subordinates 
-Need  for  technical  training  in  such  areas  as 
price/cost  analysis,  cost  estimating,  contract  admin 
istration,  cost  accounting  standards,  overhead  analy 
sis,  etc. 

Factfinding 

-What  to  look  for 

-How  to  conduct  meetings 
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Appendix  G  (Continued) 

Second  Expert  Questionnaire  Results 


Requests  for  Equitable  Adjustments 
-Rules  for  establishing  price 

-Contract  terms,  e.g.  Not-To-Exceed ,  Changes  Clause 
-Differences  between  equitable  adjustment  and  repricing 
-How  to  achieve  "leave  them  as  you  found  them"  while 
complying  with  TINA 

-Form  in  which  proposal  should  be  submitted 

Source  Selection 

-Methods  of  evaluation 
-Rul es 

-How  to  perform  cost  realism  assessment  with  limited, 
uncertified  cost/price  data 
-Define  realistic,  complete,  and  reasonable 
-How  to  handle  uncorrected  CR’s 

Fundamentals  of  Contract  Theory 
-Contract  provisions  vs  price 
-Laws 

-Contract  types 
Depot  Competition 

-Relationship  between  Cost  Comparability  Handbook  and 
Cost  Accounting  Standards 

-How  to  perform  meaningful  comparisons  of  public  and 
private  proposals 

-Overview  of  differences  in  costing  practices  among 
the  services 

Adequate  Price  Competition  for  Other  than  FFP 

-How  to  structure  evaluation  criteria  to  obtain  apples 
to  apples  comparison 
-Examples  of  contract  arrangements 

Eichleay  Formula  for  Unabsorbed  Overhead 
-Case  law  history 

-Premise  upon  which  entitlement  is  established 
-Formula  for  calculation 
-Burden  of  proof 

-Allowability/unallowability  of  variable  and  semi¬ 
variable  costs  in  overhead  pool 


G-3 


Appendix  H 

The  Problem  of  m  Rankings:  An  Intuitive  Explanation 

Maurice  G.  Kendall  developed  and  explained  this  method 
of  measuring  the  strength  of  agreement  or  "coefficient  of 
concordance"  among  a  number  of  rankings  of  a  list  of  items. 
The  following  is  an  intuitive  explanation  of  what  is  being 
measured  and  its  usefulness. 

The  coefficient  of  concordance  (W)  is  determined  by  the 
ratio  of  the  actual  deviations  from  the  mean  of  the  rankings 
to  the  maximum  possible  deviations  from  the  mean  of  the 
rankings  ( V^aximum^  •  Accordingly,  a  "1"  would  indicate 
perfect  agreement,  while  a  "0"  would  indicate  no  or  minimal 
agreement;  a  fraction  approaching  ”1"  indicates  an  increasing 
degree  of  agreement.  An  adjusted  coefficient  of  concordance 
follows  the  Chi-squared  distribution;  hence,  the  probability 
of  a  particular  W  occurring  by  random  chance  can  be  deter¬ 
mined.  The  formulae  referenced  in  Chapter  IV  are  displayed 
below.  MS  =  V2where  MS  is  the  mean  of  the  sums  of  the 

individual  rankings  for  each  item,  m  is  the  number  of  rank¬ 
ings  (or  in  this  thesis,  the  number  of  experts),  and  n  is  the 
number  of  items  being  ranked.  S  =  Z[MS-Zj^]^  where  S  is  the 
sum  of  the  squares  of  the  differences  between  MS  and  ,  and 
£j^is  the  sum  of  the  rankings  of  each  individual  item.  W  = 
[12S]/[m^(n  -n)]  where  W  is  the  coefficient  of  concordance. 

X2  =  m(n-l)W  ,  V  =  n-1  where  =  Chi-squared  and  v  =  the 
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degrees  of  freedom.  Chi-squared  is  the  distribution  used  to 
determine  the  probability  of  a  particular  W  occurring. 

Outlined  below  is  a  simplified  illustration  of  the 
concept  involved  in  Kendall's  coefficient  of  concordance. 
Assume  two  individuals  have  ranked  a  list  of  five  items  as 
shown  in  Table  H.l. 


Table  H.l 

Example  of  Perfect  Agreement 


Item 

A 

BCD 

E 

Billy 

5 

4  3  2 

1 

Shawn 

5 

ill 

1 

Sum 

10 

8  6  4 

2 

MS 

6 

6  6  6 

6 

S 

16 

+ 

4  +  0  +  4 

+16  =40 

In 

the 

above 

case 

Billy  and  Shawn  agree  perfectly.  The 

mean  of 

the 

sum  of 

the 

rankings  is  six  ( 

the  sum 

of 

the  squares 

of  the  differences 

between  the  mean 

and  the 

sum 

of  the 

ranks  of  each  item  is 

maximized  at  forty 
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Hence,  the 


[{10-6)2  +  (3-6)2  ^  (6-6)2  (4-5)2  ^  (2-5)2] 

coefficient  of  concordance  W  is  one  (^^/4') 

Now  consider  the  example  in  Table  H.2  where  the  two 
individuals  have  minimal  agreement. 

Table  H.2 

Example  of  Minimal  Agreement 


I  tern 

Billy 

Shawn 


A  B 

1  2 

5  4 


3 


D 


Sum 

MS 


6 

6 


6 

6 


6 


S 


0 


+ 


0  + 


0  +  0  +  0 


In  this  case  the  Billy  and  Shawn  have  minimal  agreement,. 
The  mean  of  the  sum  of  the  rankings  is  still  si;:,  buu  in  tni- 
case  the  sum  of  the  rankings  of  each  item  is  also  six. 

Hence,  the  sum  of  the  squares  of  the  differences  between  the 
mean  and  the  sum  of  the  ranks  of  each  item  is  zero  and  the 
coefficient  of  concordance  W  is  zero  (^/4o) 

As  one  can  see,  perfect  agreement  results  in  a  coeffi¬ 
cient  of  concordance  of  one,  while  the  lack  of  agreement 
;esults  in  a  coefficient  of  concordance  of  zero. 
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Appendix  I 

Executive  Pricing  Course  Summary  Letter 

DEPARTMENT  OF  THE  AIR  FORCE 

AM  UNNERSirv 

AM  rotice  (MTTTUTC  OF  TECHNOLOOV 


FMH:  AFIT/LSQ  (J.  Daneman  DSN  785-7777  ext.  3261) 

2950  P  Street 
WPAFB  OH  45433-7765 

SUBJECT:  Executive  Pricing  Course  Survey  Summary 

TO:  [addrescode]  ([firstname]  [lastname]  [commphone]) 


Thank  you  for  assisting  in  the  survey  designed  to  update  the 
"Executive  Cost  and  Price  Analysis"  course.  Your  resoonses, 
along  with  those  of  the  other  respondents,  have  been  tallied 
and  ranked,  and  the  comments  summarized.  The  results  are 
enclosed.  I  recognize  the  very  busy  schedule  you  have  and  do 
appreciate  very  much  the  time  and  effort  you  have  made  in  our 
behalf . 

I  am  currently  in  the  process  of  submitting  a  revised  course 
curriculum,  based  upon  this  research,  to  the  Defense 
Acquisition  University  for  their  review  and  approval.  A 
similar  course  to  be  taught  in  AFlT’s  graduate  program  is  also 
planned.  We  hope  the  resultant  courses  will  more  adequately 
address  the  needs  of  the  pricing  managers  and  executives  who 
are  required  to  take  the  course. 

If  you  have  any  questions,  you  can  contact  me  at  DSN  785-7777 
ext.  3261. 


Jeff  Daneman,  Course  Director  Atch 

Air  Force  Institute  of  Summary  of  Survey  Results 

■technology 
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Summary  of  Survey  Resu ] t s 


In  rank  order,  from  most  desirable  to  least  desirable,  the 
recommended  tooics  for  the  Executive  Pricing  Course  are: 


.icert've  Ar'‘3rgenients 

2.  Currsri:  ar:  .ans,  C5;;rt  Cases.  ieaL;2a:‘:rs 

3.  Eti-ca:  Cor:::: 

Source  Seiecfor  Suooo'C 

a  :: 3*  * C * ' 3*' ■  ^ a  ^  *^aT*f*^  ^J00^*'**"Ce3 

5.  io'es  aro  i5s:ors'b'''t';3s  o*  ^'"ce'S  ■' 
'se3o::a::o''.s 

5>  ^sc***^c*r. 3 

5.  Aoejuats  ^r’ce  CoToevcIor  *or  Ot.ie.'  tear 
Corc'ac’.s 

Dare*  ^r*»**a*  ' *  ’ 

VWV«  VW>il*>W«  <  • 

'I.  iiesuescs  ’O'"  ;:.::a:‘e  A:2us;::er;s 
'2.  ist-Tac-rg 


*  •  V  M  '  a 

■i.  Cve'-eac  iJia: :owa:‘esi 

'5.  S:e:’a'.’:e:  Reviews,  e.g..  S"::':  C:;: 


* ■-’'CiTsrzal s  3'  CcM*rsct  ’*scrv 
Cost  Accc-'^tif'c  Stircoros 


'3. 

Ccr.Tsrc  *  i 

VA*»  A  ay  a  *  -  s  ’  'f? 

'3 

.  .2 

r  9  C  ^  ^ 

3^**  ••-r 

Esc'aces 

/s  Actua's 

•i ' 

DoD  i>r3--t 

Policy 

22. 

Cor.:ra:::r 

J'rsr**** 

A  * 

'  5 

•  «  > 

•*rA  6*Y  * 

oriuia  ’C' 

rergQrAft'w  '^Yftrra;* 

2i. 

'ca::  ar: 

Cese  :*■  C 

^33  vc::  a-- 

VWv«V>  #  >1  «'vV-'>  vvv. 

2:.  Cos:  '‘s"'::r:rc  Revews 


C omments/Sub to d i _ p_n  . Too  T e n_ T op i c s 

This  section  orovides  a  discussion  and  a  list  of 
recommended  subtooics  for  the  higner  ranked  topics.  The  term 
"polarized  is  used  and  can  be  definec  as  fellows:  A 
"polarized"  topic  is  said  to  exist  whenever  at  least  two 
respondents  ranked  the  topic  in  the  first  cuartiie  and  at 
least  two  ranked  it  in  the  last  cuartiie.  This  polarization 
of  opinions  on  some  issues  is  expected  and  reflects  valid 
observations  or  needs  based  upon  the  respondents  locations 

This  polarization  does  not  affect  the 
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and/or  experiences 


overall  ranking  of  the  tooics,  biJt  dees  af-^ect  the  acoare-'t 
strength  of  tne  agreement  or  degree  o^  consensus.  WTohout 
adjusting  for  the  polarized  topics,  the  data  reflects  a  67% 
level  of  confidence  that  the  ranking  reflects  a  consensus. 
After  adjusting  for  four  polarized  issues,  the  level  of 
confidence  rises  to  88%,  which  is  considered  adeouate  for 
decis'ons  regarding  "issues  as  general  as  educat'o~. 

T.?p1.c.J  The  ooints  of  emDhas"'3  cr  subtocics  i-eccmmercec  for 
the  discussion  of  incent"ive  arrangements  included  -"ev^ews  of 
cost  and  fixed  price  incentive  con"cracts,  oerformance 
incentives,  working  with  multiple  "incentives,  and  ocssTo’e 
trade-offs . 


Topic  2  A  review  of  current  and  pending  laws,  r egu 1  at ' ons . 
and  board  or  court  cases,  including  changes  in  tne  Feceral 
Acquisition  Regulations  and  Deoart.ment  of  Defense  *SGer-.' 
Accuisition  Regula"tion  Suoplem.ent.  shoulo  emphasize  new 
"zrends  and  developments  as  opposed  to  administrat've 

Topic  3  (also  #71  The  comments  regarding  ethical  concuct 
were  very  closely  related  to  comments  relating  to  etrmcal 
settlements  in  negotiations.  Accordingly,  the  subtopics  and 
points  of  emphasis  will  be  listed  together.  .\c  comments 
referenced  Air  Force  Regulation  30-30  or  sim.iiar  ethics 
regulations  or  credos.  The  subtopics  include  concepts  such 
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as  fair  and  reasonable  sett  1  enrients  and  open  communications. 


Specifically,  the  listed  subtooics  were  equity  for  both 
government  and  contractor,  what  constitutes  a  settlement, 
what  ethics  are  involved  in  negotiations,  how  to  elicit 
compliance  with  the  Truth  in  Megotiations  Act  and  the 
disclosure  of  unallowable  costs,  and  "does  'law'  protect  the 
government  from  poor  ethics  on  contractor's  part?". 

Topic  4  The  topic  of  pricing  suopcrt  for  source  selection 
should  include  the  fol ''owing  subtoo'cs:  methods  of 
evaluation:  defining  realistic,  complete,  and  reasonable:  how 
to  oerform  cost  realism,  assessments  with  li.mited.  ...ncs'' t  i  f  '  ed 
cost  and  pricing  data;  how  to  nandle  uncorrected 
Deficiency  Reports  (DR's);  and  the  rules  of  source  selection. 

Topic  5  To  establishing  a  training  program  for  subordinates, 
technical  training  in.  such  areas  as  ?rice/cost  analysis,  cost 
estimating,  contract  administration,  cost  accounting 
standards,  overhead  analysis,  and  so  forth  must  be  addressed. 

Topics  6  (also  tf19)  The  topic  of  roles  and  responsibilities 
of  pricers  in  negotiations  is  closely  related  to  the  topic 
which  ranked  nineteenth  overall,  program  management/or i c i ng 
conflict  resolution,  and  which  is  polarized.  What  is  being 
sought  is  a  clearer  definition  of  the  .mission  and  role  cf  the 
pricer,  the  contracting  officer,  the  auditor,  other  technical 
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support  persor.ne',  and  o~  the  program  manager .  ’I’hose  ro'es 
are  defined  by  each  particular  office’s  organ  i  zac :  or.a ' 
structure  and  cultural  norms.  For  example,  one  ourchasing 
center  reportedly  exhibits  conflict  between  the  program 
managers  and  the  pricers,  whereas  such  conficts  seldom 
ever  occur  at  other  purchasing  centers.  The  norms  o~  the 
'  atter  o'^fices  have  more  clearly  defined  roles  ^or  eacr 
function.  !n  addition  to  cleare-  de-^  '  n  i  t  i  or  s  .  tecrrioues  a 
needed  to  resolve  conflicts  v/henever  thev  occur. 


:pic  7  See  too  ’  c 


Tppjc.,8  There  were  polarized  views  ’'egard*'*?  w-et'-e-  t-e 
tooic  of  factfinding  should  be  considered  fcr“  an  execut've 
or  icing  cource.  On  the  "nc"  side  are  those  wnc  'ee'  t'.a* 
eitner  the  pricer  is  just  one  cf  a  factfinding  team  (not  t' 


eacer ;  or  tnat  factr’raTng  is  a  oasic  tasK  a 
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oy  all  who  are  recuired  to  take  the  course.  On  t:“e  "■'■es" 
s'de  are  those  who  feel  that  the  pricer  is  the  kev  o'ave-  ■ 
factfinding  and  that  inadequate  attention  has  been  g-ven  to 
ensuring  that  all  elements  of  the  statem.ent  of  work  are 
addressed  and  aoproor  i  ate  1  y  priced  in  the  cont-actcr" ' s 
proposals.  Accordingly,  the  recommended  subtcoics  are  what 


to  look  for  and  how  to  co:  duct  the  meetings 


an 


Topic  9  Obtain -ing  adecuate  price  ccmoet i t i on  'cr  otner  -n 

firm  fixed  price  contracts  requires  structuring  the 
evaluation  criteria  to  obtain  an  "apo ; es-to-apo  i  es" 
comparison.  Requested  are  examples  for  fixed  price  incentive 
firm,  cost  plus  incentive  fee,  cost  plus  fixed  fee,  and  time 
and  material  contract  types. 

Topic  10.  Some  pricing  offices  have  become  involved  in  the 
evaluation  cf  competitive  proposals  submitted  by  various 
depots  and,  sometimes,  private  entities.  To  evaluate  the 
proposals,  a  better  understand i ng  of  the  relationship  between 
the  Cost  Comparab 7 ? i ty  Handbook  and  the  Cost  Account -ng 
Standards,  instruction  on  how  to  perform  meaningful 
comparisons  of  public  and  private  proposals,  and  an  overview 
o*  differences  in  costing  practices  among  the  services  are 
needed . 


Oth_er  Po  1  ar  i  zed__  Op  i  r  ,i_ons 

The  eighth  and  nineteenth  topics,  discussed  above,  are 
polarized.  The  eleventh  topic,  requests  for  equitable 
adjustments,  was  another  topic  where  polarized  opinions  were 
exhibited.  in  smaller  supply  contracts  such  requests  usually 
involve  simpler  issues  which  are  easily  resolved.  in  major 
acquisitions  requests  for  equitable  adjustments  may  be  quite 


complex  wipr.  mar.v  issues,  ap.d  may  be  an  atberrob  'to  reorice 
the  contract  after  a  "buy-in.”  Acccrbing'y,  the  r ecommendec 
subtooics  include  how  to  di~ferer.r.iate  oetween  ecu  it  able 
adjustments  and  repricing,  and  how  to  interpret  arc  apply 
various  contract  provisions  such  as  the  Changes  Clause  or  a 
Not-to-Exceed . 


~he  sixteenth  tcoic,  the  fundamentals  contract 
was  also  a  polarized  topic.  The  fundamentals  cf 
theory  would  address  the  basics  of  contracting  fc 
price  analysts  who  have  had  no  contracting  exoer- 
of  concern  are  contract  provisions  versus  orice  a 
apoli cable  laws,  and  the  dif'^erences  among  contra 
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CP.T'.Dl  e.O.t  s 

The  twenty-tr  r  d  tooic.  tre  Eic^le.n-/  "ormula  'l^cr  .:':abcc ''oec 
overhead,  was  ranked  in  the  too  five  b^■  at  'east  ore 
individual  and,  therefore,  received  comments.  ~he  E'’c''''eay 
formula  for  unabsorbed  overhead  wou'd  recuire  net  only 
coverage  of  the  techniaue  itself,  but  also  the  case  law 
history  and  the  orem.ise  uocn  which  entitlement  is  based. 
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Appendix  J 

Description  of  Topics 

Incentive  Arrangements .  This  topic  involves  two  general 
areas.  The  first  is  technical:  How  to  price  incentive 
contracts  involving  multiple,  conflicting  incentives,  e.g., 
increase  performance  and  decreased  cost,  or  similar  complex 
arrangements.  The  second  area  involves  identifying  and 
addressing  those  areas  that  truly  incentivise  the  contractor, 
those  areas  that  are  of  greatest  to  the  contractor  such  as 
positive  long-term  relationships,  internal  politics,  or  the 
reduction  of  risk. 

Current  and  Pending  Laws ,  Court  Cases .  and  Regulations . 
This  review  should  emphasize  new  trends  and  developments  as 
opposed  to  administrative  or  procedural  changes. 

Sthical  Conduct  and  Ethical  Settlements  in  Negotiations . 
The  topic  ethical  conduct  is  very  closely  relatt-d  to  the 
topic  ethical  settlements  in  negotiations.  Accordingly,  th-y 
will  be  described  together.  Notably,  the  topics  do  not  refer 
to  Air  Force  Regulation  30-30  or  similar  ethics  regulations 
or  credos.  Instead,  they  refer  to  concepts  such  as  fair  and 
reasonable  settlements,  open  communications,  and  equity  for 
both  government  and  contractor. 

Source  Selection  Pricing  Support .  Pricing  support  for 
source  selection  should  involves  determining  that  proposed 
costs  are  realistic  (which  requires  a  comparison  of  the 
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statement  of  work,  work  breakout  structure,  and  cost  buildup), 
complete,  and  reasonable;  at  times  involves  using  limited, 
uncertified  cost  and  pricing  data;  and  has  limited  and 
highly  structured  communication  mechanisms. 

Establishing  a  Training  Program  for  Subordinates .  A 
training  program  for  pricers  should  interrelate  on-the-job 
training,  the  required  professional  continuing  education 
courses,  and  other  formal  courses  and  informal  training 
opportunities  that  are  designed  to  enhance  the  individuals 
overall  capabilities  and  groom  them  for  management  opportuni¬ 
ties.  To  establishing  a  training  program  for  subordinates, 
technical  training  in  such  areas  as  price/cost  analysis,  cost 
estimating,  contract  administration,  cost  accounting  stand¬ 
ards,  overhead  analysis,  and  so  forth  must  be  addressed. 

The  Roles  and  Responsibilities  of  Pricers  in  Negotia¬ 
tions  and  Program  Management,' Pricing  Conflict  Resolution . 

What  is  needed  in  a  discussion  of  these  topics  is  a  cleare. 
definition  of  the  mission  and  role  of  the  pricer,  the  con¬ 
tracting  officer,  the  auditor,  other  technical  support  per¬ 
sonnel,  and  of  the  program  manager.  Those  roles  are  defined 
by  each  particular  office's  organizational  structure  and 
cultural  norms.  For  example,  one  purchasing  center  reported¬ 
ly  exhibits  conflict  between  the  program  managers  and  the 
pricers,  whereas  such  conflicts  seldom  if  ever  occur  at  other 
purchasing  centers.  The  norms  of  the  latter  offices  have 
more  clearly  defined  roles  for  each  function.  In  addition  to 
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clearer  definitions,  techniques  are  needed  to  resolve  con¬ 
flicts  whenever  they  occur. 

Factfinding .  In  some  organizations  the  pricer  is  the 
key  player  in  factfinding  and  adequate  attention  must  be 
given  to  ensure  that  all  elements  of  the  statement  of  work 
are  addressed  and  appropriately  priced  in  the  contractor's 
proposals.  Pricers  need  to  know  what  to  look  for  and  how  t 
conduct  the  meetings. 

Adequate  Price  Competition  for  Other  than  Firm  Fi xed 
Price  Contracts .  In  some  organizations  pricers  are  tasked 
with  establishing  guidelines  for  structuring  the  evaluation 
criteria  fcr  determining  the  winner  in  price  competitions 
involving  fixed  price  incentive  firm,  cost  plus  incentive 
fee,  cost  plus  fixed  fee,  and  time  and  material  contract 
types.  Developing  criteria  that  yields  an  "apples  to  apple 
compar~son  is  at  times  very  difficult. 

Depot  Competi tion .  Some  pricing  offices  have  became 
involved  in  the  evaluation  of  competitive  proposals  submitt 
by  various  depots  and,  sometimes,  private  entities.  To 
evaluate  the  proposals,  a  better  understanding  of  the  rela¬ 
tionship  between  the  Cost  Comparability  Handbook  and  the  Co 
Accounting  Standards,  instruction  on  how  to  perform  meaning 
ful  comparisons  of  public  and  private  proposals,  and  an 
overview  of  differences  in  costing  practices  among  the  serv¬ 
ices  are  needed. 
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Requests  for  Equitable  Adjustments .  In  smaller  supply 
contracts  such  requests  usually  involve  simpler  issues  which 
are  easily  resolved.  In  major  acquisitions  requests  for 
equitable  adjustments  may  be  quite  complex  with  many  issues, 
and  may  be  an  attempt  to  reprice  the  contract  after  a  "buy- 
in."  Accordingly,  instruction  would  include  how  to  differen 
tiate  between  equitable  adjustments  and  repricing,  and  how  t 
interpret  and  apply  various  contract  provisions  such  as  the 
Changes  Clause  or  a  Not- to-Exceed. 

Fundamentals  of  Contract  Theory .  The  f  unda.mental s  cf 
contract  theory  would  address  the  basics  of  contracting  for 
those  price  analysts  who  have  had  no  contracting  experience. 

The  Eichl eav  Formul a  for  Unabsorbed  Overhead .  The 
Eichleay  formula  for  unabsorbed  overhead  is  occasionally  use 
in  cases  that  is  in  litigation.  Coverage  would  include  the 
technique  itself,  the  case  law  history,  and  the  premise  up^a 
which  entitlement  is  based. 

Estimating .  Usually  pricers  aie  not  involved  in  de¬ 
veloping  estimates,  rather  reviewing  estimates  made  by  eth¬ 
ers.  Occasionally,  such  as  in  source  selection,  esti.r.ates  b 
the  pricer  are  required.  Accordingly,  the  techniques  cf 
establishing  realistic  and  reasonable  estimates  are  needed. 

Claims .  Pricing  a  claim  is  similar  to  other  pricing 

action  except  claims  occasionally  require  consideration  for 

« 

burden  of  proof.  The  policy  stated  in  the  Federal  Acquisi¬ 
tion  Regulations  is  that  claims  should  be  negotiated  if 
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possible.  Accordingly,  some  pricers  need  to  learn  to  see 
the  big  picture  and  the  contractor's  point  of  view,  and  have 
less  emphasis  on  individual  elements  of  cost. 

Overhead  ( Unal lowabl es ) .  This  topic  involves  the  review 
of  FAR  part  30,  "Cost  accounting  standards"  and  FAR  part  31. 
Specifically  addressed  is  how  to  identify  and  account  for 
unallowable  overhead  costs. 

Specialized  Reviews .  a . a . ,  Should  Cost .  This  topic 
would  review  the  principles  and  techniques  involved  in  should 
cost,  could  cost,  pre-award,  and  similar  reviews. 

Cost  Accounting  Standards .  This  is  a  more  detailed 
review  of  the  cost  accounting  standards  published  in  FAR  pan 
3C  and  the  changes  occurring  as  a  result  of  the  Chief  Finan¬ 
cial  Officers  Act  of  1990  and  similar  legislation. 

Pricing  of  Commercial ,  Nondevel opmental  Items.  This 
topic  would  cover  techniques  required  to  ensure  that  prices 
paid  for  commercial,  nondevel opmenta 1  iiems  arc  fair  and 
reasonable.  Related  to  this  topic  is  how  to  perform  a  vein- 
analysis  . 

Estimates  vs  Actuals.  This  topic  covers  techniques  o 
comparing  contractor's  estimates  to  incurred  costs  for  the 
purpose  of  negotiating  an  estimate  to  complete.  Involved  is 
learning  how  to  work  with  less  than  ideal  data. 

DoD  Profit  Policy.  ^  This  topic  would  deal  with  current 
profit  policy,  the  intent  of  congress,  and  the  application  of 
policy  to  various  contract  types. 
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Contractor  Financing .  This  topic  involves  a  review  of 
the  various  types  of  contractor  financing,  its  impact  on 
contract  type  and  profit,  and  its  impact  on  special  cases 
such  as  contract  in. litigation. 

Impact  and  Cost  of  C/SCSC ,  CPR ,  C/SSR .  etc .  These  types 
of  reviews  are  often  required  but  are  not  priced  separately. 
Consequently  we  do  not  understand  their  impact  on  price  or 
upon  the  contractor's  regular  information  reporting  system. 
This  topic  would  discuss  their  impact  and  viable  alternative 
tailored  requirements. 

Cost  Monitoring  Reviews .  This  topic  concentrates  upon 
the  impact  of  and  techniques  required  to  accomplished  cost 
reviews . 
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